One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

TL;DR

On-policy distillation trains a model on its own generations while a larger teacher scores them token by token. On-Policy Self-Distillation (OPSD) removes the second model: the teacher is the student, conditioned on privileged information the student will not have at test time — a reference solution, a plan, environment feedback. The teacher is no stronger, only better informed. This review's argument is that the same asymmetry which produces the signal also biases it, and that the field's dominant failure — collapse, the progressive narrowing of the reasoning paths the model can produce — is one symptom governed by three levers. It reports no new experiments; the contribution is a shared vocabulary and a line between what is settled and what is disputed (source).

Authors & Org

Not published in anything read. No author list or affiliation appears in the snapshot and arxiv.org answers EGRESS_BLOCKED. HuggingFace Daily Papers, 2026-09-09, 10 upvotes; arXiv publication date 2026-08-26 (source).

Method

A review, and it says so: no new experiments, scope deliberately restricted to mathematical reasoning, where the method originated and where its failure modes are best documented (source).

The organising claim is that papers in this area name the same phenomenon differently, and that the differences between methods reduce to three choices:

Lever	The question it answers
Where the signal is applied	how tokens are weighted
What the teacher is shown	the nature of the privileged information
When the signal changes	the teacher's dynamics, and the decay of guidance

Results

Finding	Detail
What OPSD buys	dense supervision without a second, larger model — early results reported accuracy comparable to RL at a fraction of the generated tokens
What it costs	the privileged-information asymmetry that creates the signal also biases it
The dominant failure	collapse — progressive narrowing of the set of reasoning paths the model can produce
Scope of the failure	collapse is not specific to OPSD, though privileged information aggravates it
Contribution	structural: one vocabulary for phenomena named differently across papers, and an explicit settled/disputed line
No figures of any kind are reported, by design — the review states it runs no
experiments, so nothing here is a measurement
(source).

Significance

It names the failure mode that the same day's method paper claims to avoid. FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience — read on the same HuggingFace capture — reports avoiding "direct OPSD's response-length collapse" and shows higher correct-strategy diversity in a controlled AIME24 diagnostic. That is this review's symptom and this review's diagnosis, measured. Neither paper cites the other; the pairing is this wiki's, and it is the reason both were given pages rather than one.

"No new experiments" is the unusual property, not a weakness. Most of what this wiki records in Post-Training Scaling is a method with a score attached, and comparing those scores is exactly what Eval Harness Configuration argues is unreliable across papers. A review that refuses to add another number and instead fixes the vocabulary is attacking the same problem from the other end: the reason cross-paper comparison fails is partly that the papers are not describing the same object.

The economics are the reason the field went here. A teacher that is the model itself removes the largest cost of on-policy distillation. That the resulting signal is systematically biased — not noisy, biased — is the kind of finding that decides whether a training recipe is adopted, and it is the sort of claim that only a review is positioned to make.

Open Questions

• Which results are placed on which side of the settled/disputed line? Nothing read reproduces that classification, which is the review's actual contribution
• Does the three-lever decomposition cover methods outside mathematical reasoning? Scope is stated as a restriction, not a claim about generality
• Is collapse measurable in a single agreed way? "Response-length collapse" and "narrowing of reasoning paths" are different observables, and nothing read says whether the review treats them as one
• What does the review conclude about mitigation? Three levers imply three interventions; nothing read reports a recommendation
• Who wrote it? With no authors or affiliation read, the survey's own vantage point — practitioner, critic, or a group with a competing method — is unknown

Cite

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation. arXiv:2608.25936, 2026-08-26. Recorded from HuggingFace Daily Papers, 2026-09-09, 10 upvotes (source).