$ cat wiki/papers/2026/2605.15155-self-distilled-agentic-rl.md
Self-Distilled Agentic Reinforcement Learning
paperupdated 2026-08-14created 2026-05-16
arxiv: 2605.15155 Authors: 11, first author Zhengxi Lu Submitted: 2026-05-14 · captured from HF Daily 2026-05-16 (#3, 75 upvotes)
TL;DR
SDAR adds token-level distillation guidance to reinforcement learning for multi-turn agents: a teacher model's signals pass through a sigmoid gate in an auxiliary objective, amplifying helpful corrections and softening unhelpful rejections. Reported as 7–10% gains over a GRPO baseline on three agent benchmarks (source).
Method
The self-distillation is a shaping term on the RL objective, not a separate pre-training stage (source):
| Element | As described |
|---|---|
| Base algorithm | reinforcement learning over multi-turn agent trajectories |
| Added signal | token-level distillation guidance from a teacher model |
| Gating | a sigmoid gate in an auxiliary objective |
| Effect of the gate | amplifies helpful corrections, softens unhelpful rejections |
| Gating at the token level is what distinguishes it from distilling a whole | |
| trajectory: the teacher can be trusted on part of a turn and discounted on the | |
| rest. |
Results
Improvements over a GRPO baseline on the same setup (source):
| Benchmark | Δ vs GRPO |
|---|---|
| ALFWorld | +9.4% |
| Search-QA | +7.0% |
| WebShop (accuracy) | +10.2% |
| The abstract reports no absolute scores. Every figure here is a delta against | |
| GRPO, so quoting one without its baseline states something the paper does not. |
Significance
- Matches this wiki's weighted interests twice over — Agentic Reinforcement Learning (1.5x) and RL (1.3x) — which is why it was surfaced
- HF Daily #3 on 2026-05-16 with 75 upvotes: that community's popularity signal on the day, not a quality judgement
- If a gated auxiliary term buys 7–10% over GRPO, the interesting claim is that agentic RL is limited by credit assignment within a turn rather than by the amount of trajectory data
Open Questions
- Three benchmarks, all deltas against one baseline. How does SDAR compare against methods other than GRPO?
- Where does the teacher come from, and what happens when it is no stronger than the student — does the gate degrade to plain RL?
- Stability across longer horizons, and catastrophic forgetting, are untouched by the abstract
Cite
@article{self_distilled_agentic_rl_2026,
title={Self-Distilled Agentic Reinforcement Learning},
year={2026},
eprint={2605.15155},
archivePrefix={arXiv}
}