$ cat wiki/papers/2026/2608.14277-simpleopd.md
SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning (arXiv:2608.14277)
TL;DR
On-policy distillation across tokenizers: align only the tokens occupying
identical text spans under teacher and student tokenizers, distil in a shared
text space, and control length explosion with a student reference KL loss
plus masking the advantages of termination tokens (</think>, <|im_end|>).
Transferring proof reasoning from the long-context model SU-01, the headline
is Intern-S2-Preview +21.2 points on ProofBench, to 55.2, stated as surpassing
Gemini-2.5-Pro, with gains also on HLE and HiPhO
(source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only
(source).
Listed on HuggingFace Daily Papers, 2026-08-18, 23 upvotes — that community's popularity signal and nothing more (source).
Method
Four named obstacles to applying on-policy distillation from a long-context reasoning teacher to a short-context student (source): tokenizer mismatch, teacher–student distribution mismatch, response length explosion, and training instability.
| Obstacle | Stated remedy |
|---|---|
| Tokenizer mismatch | distil in a shared text space; align only tokens occupying identical text spans under both tokenizers |
| Length explosion / truncation | student reference KL loss, constraining drift from the initial policy |
| Length explosion / truncation | mask the advantages of special termination tokens — </think>, <|im_end|> |
| Distribution mismatch | the same KL constraint, described as fostering steady length growth rather than a jump |
| The teacher is SU-01, a long-context reasoning model. Students span | |
| same-family and different-family models: **Qwen3, Qwen3.5, Intern-S2, | |
| GLM-4.7, Gemma-4**. |
Results
| Measure | As reported |
|---|---|
| Intern-S2-Preview on ProofBench | +21.2 points, reaching 55.2 |
| Stated comparison | surpasses Gemini-2.5-Pro on that benchmark |
| Other benchmarks | gains on HLE and HiPhO |
| Breadth | "consistent gains" in mathematical reasoning across the five student families, "especially natural-language math proving" |
| The generalisation claim is the paper's own and is narrow in its wording: the | |
| science-benchmark gains are offered as evidence that OPD transfers reasoning | |
| "beyond the mathematical training domain". |
What the abstract does not give: any figure except the ProofBench pair — no per-student numbers, no HLE or HiPhO values, no baseline scores, no context lengths, no cost, and nothing about SU-01 beyond the label "long-context reasoning model".
Significance
It is a measured downstream number for a model this wiki holds. Intern-S2-Preview: Scientific Agentic Foundation Model (arXiv:2608.13505) was written from its own abstract on 2026-08-17. A day later a different paper reports moving that checkpoint +21.2 points on ProofBench — a third-party figure on a first-party artefact, which is the pattern this wiki keeps asking for and rarely gets on a paper-to-paper timescale. It also means Intern-S2-Preview appears in three different roles across two days: a released model, a student here, and, in Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning (arXiv:2608.14290), a series whose successor rebuilds the backbone instead.
On-policy distillation shows up twice in one 25-paper batch. The other is Self-Supervised Visual On-Policy Distillation (S²VOPD, arXiv:2608.14144, 146 upvotes), which attacks the same dependency from the opposite side: rather than finding a stronger teacher, it subtracts information from the student — distilling the teacher's distribution on the original image into a student seeing a strongly augmented view — and reports Qwen3.5-4B improving 70.7% → 77.4% across six fine-grained perception benchmarks, recovering 96% of what privileged-information methods achieve (source). Both papers treat teacher–student asymmetry as the scarce resource and manufacture it cheaply — one across tokenizers, one across augmentations. They do not cite each other; the pairing is this wiki's and is labelled as such.
The termination-token detail is the transferable finding. Masking the
advantage on </think> and <|im_end|> says the student was learning when to
stop from a teacher whose stopping behaviour is calibrated to a much longer
context — a mechanism, not a hyperparameter, and directly relevant to
Test-Time Compute (Inference-Time Compute Scaling): it is a fourth thing in two days that changes how
many tokens an answer costs, alongside reallocation, architecture and refusal.
Open Questions
- What is SU-01? The teacher is the source of every gain reported and nothing read identifies its developer, size, context length or availability. A result that cannot be reproduced without it is a result about an unavailable artefact.
- One number, five families. "Consistent gains" across Qwen3, Qwen3.5, Intern-S2, GLM-4.7 and Gemma-4 is supported in the abstract by exactly one student's one benchmark.
- ProofBench 55.2 against Gemini-2.5-Pro — a comparison against a model more than a year old at time of writing, not against a current frontier model, and the abstract does not say why.
- What does the identical-text-span rule discard? Aligning only spans that match under both tokenizers necessarily drops the rest; no coverage figure was read.
- Author list, affiliation, code — unknown; the paper was not read.
Cite
SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context
Reasoning (2026). arXiv:2608.14277.