When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

TL;DR

Student models distilled on-policy from post-trained teachers produce excessively long responses that can exhaust the generation budget, and a substantial part of the cause is that teacher and student put their stopping probability on different end-of-sequence tokens — even when their declared stopping sets are identical (source).

Authors & Org

Not stated. The HuggingFace Daily Papers snapshot carries the arXiv id, title, upvote count, publication date and abstract; it carries no author list and no affiliation, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox. Recorded as unknown rather than guessed.

Published 2026-09-17, 75 upvotes in the 2026-09-20 snapshot — that community's popularity signal and nothing more (source).

Method

The failure is located at the termination action rather than in the loss, the reward, or the length penalty — the three places length inflation is normally attributed to.

Element	What the paper does
Setting	on-policy distillation (OPD): base student, post-trained teacher
Model families	Qwen3, Llama, Gemma
Diagnosis	teacher and student place stopping probability on different EOS tokens, despite identical declared stopping sets
Intervention tested and rejected	aligning the decoding stopping set alone — "insufficient"
Intervention that works	treating functionally equivalent EOS tokens as a shared semantic stopping action
Second experiment	OPD across different K2-Horizon training stages, to watch termination preference move during training
(source)

Results

1. The mismatch suppresses without substituting. The teacher's signal can suppress the student's preferred termination action "without reliably transferring the teacher-preferred alternative" — so the student is left with no confident way to stop, which is the mechanism behind the inflation.
2. Fixing the declared set is not enough. Aligning the decoding stopping set alone is stated to be insufficient; the equivalence has to be treated as semantic, not as a list.
3. The fix generalises across families. Treating functionally equivalent EOS tokens as one shared stopping action substantially mitigates mismatch-induced length inflation across all three model families.
4. It is not the whole story, and the paper says so. The stage-wise analysis finds termination preferences shift substantially during training, and a distinct length inflation late in the OPD run that persists beyond termination alignment. The authors' own summary: termination mismatch is "an important, but not exhaustive, source of OPD length dynamics."

An implementation with the termination-handling corrections is released (source).

Significance

It is a tokenizer bug wearing an RL bug's clothes. Length inflation in distilled students is normally treated as a reward-shaping or objective problem. This locates a large part of it in an artefact of vocabulary design — two models that agree on what the stopping set is and disagree on which member of it carries the probability mass. The declared set is identical, so every check that compares stopping sets passes.

That makes it a sibling of the result Eval Harness Configuration exists for. Both are cases where a component nobody names as a variable turns out to carry the effect, and where the reported number is a property of the plumbing rather than of the method. Here the plumbing is the vocabulary.

The honest part is the residual. The late-training inflation that survives the fix is reported rather than left out, which is what makes the "important but not exhaustive" claim checkable by whoever reads this next.

Open Questions

• Which EOS tokens, in which families. The abstract states the mismatch occurs across Qwen3, Llama and Gemma; no token, id or vocabulary detail appears in anything read.
• What causes the late-run inflation that persists beyond termination alignment. The paper names it and does not explain it.
• How large the effect is. No figure of any kind — no length ratio, no budget-exhaustion rate, no downstream accuracy delta — appears in the snapshot's abstract, which is everything this run could read.
• What "K2-Horizon" is. It appears once, as the training-stage series the second experiment uses. Unidentified in anything read, and not assumed to be related to Kimi K3 or any other K2-named model.
• Whether the same mismatch affects RL from a teacher's logits generally, or only the on-policy distillation setting.

Cite

arXiv 2609.20511, published 2026-09-17 (arXiv) (source).

Related

• Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening — the same shape of result one layer over: a standard component of the RL recipe failing quietly
• Post-Training Scaling
• Agentic Reinforcement Learning
• Eval Harness Configuration