$ cat wiki/concepts/agentic-rl.md
Agentic Reinforcement Learning
Definition
A paradigm in which an LLM agent learns via RL while interacting with an environment. Instead of a single response, it optimizes sequences of multi-step tool use, planning, and reflection against a reward signal.
Why It Matters
- Squarely at the center of my interests: the intersection of agents + RL + reasoning
- A limitation of plain RLHF (evaluating a single response) — agents require sequence-level evaluation
- One of the core research currents of 2026
State of the Art (2026-08-20)
-
Harnessed agentic RL — the deploy-time harness now participates in training (2026-08-18): Agent Lightning v1.0: Towards Harnessed Agentic RL (arXiv:2608.17528) names the regime and ships a ~3,500-line framework for it. Arbitrary agents connect to training through an LLM endpoint proxy, so the harness owns the environment interaction loop and the trainer observes only LLM request/response pairs. The paper states four other frameworks — verl Uni-Agent, AReaL 2.0, slime, Polar — have adopted the architecture. Reported: Qwen3.5-9B 41.8% → 56.4% on SWE-bench Verified from 6K examples. Consequence: a model post-trained this way is fitted to its harness, which is the mechanism behind ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798)'s finding that such a model stops being harness-agnostic. The harness it used for the 56.4% is not named (source)
-
The competing bet: delete the credit-assignment machinery rather than fix it (2026-08-18): Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements (arXiv:2608.17310) argues evolution strategies suit long-horizon agents better than RL — full-parameter optimization at inference-level GPU memory, and trajectory-level parameter attribution with no reward decomposition across the horizon. Reported: +6.69% over a No-Skill baseline on WebArena-Lite for Qwen-3.5-27B, and its matched baseline beaten in 28 of 36 test-time heuristic-design settings. No head-to-head against agentic RL is reported, and no rollout count is published, so the efficiency claim is memory, not compute (source)
-
Anthropic AAR (2026-04-14): 9 instances of Claude Opus 4.6 worked on an alignment research problem (weak-to-strong supervision) for one week in an agentic RL setup → PGR 97% vs. 23% for human researchers. The first empirical case of RL agents replacing humans on a real-world research problem. → Automated Weak-to-Strong Researcher (AAR)
-
Self-Distilled Agentic RL (2026-05-16 HF Daily #3): Self-Distilled Agentic RL
-
Direct On-Policy Distillation / Direct-OPD (2026-07-06, Tsinghua AIR + ByteDance Seed): Run RL on a cheap weak model, then transfer only the RL-induced policy shift (
Δ = log π_T − log π_{T,ref}) to a large model using on-policy distillation. Cuts the cost of deploying RLVR gains at frontier scale. A practical cost lever for the Anthropic AAR-style workload where frontier RL is expensive. → Weak-to-Strong Generalization via Direct On-Policy Distillation -
Related prior work: ReAct, Reflexion, AutoGPT family (page TBD)
Open Problems
- Multi-step credit assignment — Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements (arXiv:2608.17310) proposes sidestepping it entirely via trajectory-level attribution, rather than solving it
- Reward hacking in tool-use environments
- Data efficiency (online RL is expensive) — Direct-OPD partially addresses via weak-model rollouts
- Long-horizon stability
Key Papers
- Automated Weak-to-Strong Researcher (AAR) — Anthropic AAR, 2026-04-14 (97% PGR, weak-to-strong supervision)
- Self-Distilled Agentic Reinforcement Learning — 2026-05
- Weak-to-Strong Generalization via Direct On-Policy Distillation — Direct-OPD, Tsinghua AIR/ByteDance, 2026-07-06 (cost-efficient RL transfer)
- SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning — SEED, 2026-07-14 (on-policy distillation for agents)
- Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning — Ring-Zero, Ant Group, 2026-07-12 (RLVR at a trillion parameters; five emergent behaviours)
- Agent Lightning v1.0: Towards Harnessed Agentic RL (arXiv:2608.17528) — Agent Lightning v1.0, 2026-08-18 (harnessed agentic RL; SWE-bench Verified 41.8% → 56.4% on Qwen3.5-9B)
- Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements (arXiv:2608.17310) — Agentic ESOpt, 2026-08-18 (evolution strategies instead of RL for long-horizon agents)