$ cat wiki/papers/2026/2608.26530-pilot.md
PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents (arXiv:2608.26530)
TL;DR
A supervisor-worker harness that updates itself during a run rather than after it: a separate supervisor can redirect or abort the active worker, and procedures and failure modes surfaced mid-run are distilled into reusable skills and memory. First in five of six configurations across two frozen backbones and three benchmarks; up to 9.8 percentage points over counterpart harnesses on Terminal-Bench 2.0; mean output tokens fall 42.9% and 47.4% while successful evaluations per million output tokens rise 110.3% and 134.0% (source).
Authors & Org
Yang Xiao, Yusong Sun, Haoyi Wu, Wenyang Hui, Wen Da, Zhaokai Luo, Mu Chuan, Yao Hu, Wenjie Li, Chengyue Jiang (arXiv:2608.26530). Submitted 2026-08-27; cs.AI. No affiliations shown on the arXiv listing.
Listed on HuggingFace Daily Papers, 2026-08-28, 15 upvotes — that community's popularity signal and nothing more (source).
Method
The stated gap: "Most self-improvement methods process this experience only after execution ends, so they cannot redirect the active run or immediately apply and validate lessons learned from it."
The paper's diagnosis of why existing architectures cannot do this: "Single-agent self-correction combines task execution and trajectory assessment within one context, while subagent delegation separates execution but typically cannot redirect an active subagent."
PILOT's two coupled mechanisms:
- Live steering — "a separate supervisor" can "redirect or abort the active worker during execution".
- Live self-evolution — "distils procedures and failure modes revealed during execution into reusable skills and memory", updating "the persistent harness".
Evaluation: two frozen backbones, three benchmarks (source).
Results
| Measure | Reported |
|---|---|
| Configurations where PILOT ranks first | 5 of 6 |
| Terminal-Bench 2.0, over counterpart harnesses | up to 9.8 percentage points |
| Self-improvement setting, GLM-5.1 | +14.6 points |
| Self-improvement setting, Kimi-K2.6 | +12.4 points |
| Mean output tokens | −42.9% and −47.4% |
| Successful evaluations per million output tokens | +110.3% and +134.0% |
| The backbones are two generations back on both sides. Kimi K3 is | |
| the current Moonshot model this wiki holds and K2.6 is what was run; Z.ai's | |
| current line is GLM-5.3 and GLM-5.1 is what was run. Nothing read | |
| states why. A harness result on older backbones is not evidence about the current | |
| ones, and the direction of the error is not knowable in advance — a better model | |
| may need less steering or may follow it further. |
Significance
This lands in the middle of an argument Eval Harness Configuration has been tracking since 2026-08-14, and it takes a side.
The token efficiency figures are the more interesting half. Every harness result on that page so far has bought accuracy with more computation — more rollouts, more repair passes, longer trajectories. PILOT reports the opposite trade: fewer output tokens and more successes per token, in the same runs. If that holds, a supervisor that aborts early is not merely a quality mechanism but a cost one, and the two have been assumed to pull against each other throughout this cluster.
The abort capability is also the first mechanism in the cluster that addresses FrontierChallenge: Evaluating Scientific Workflow Completion (arXiv:2608.24979)'s finding directly. That paper measured 75.5% of non-passing Claude Code trajectories ending with a claim of completion; a worker that cannot assess itself in its own context is exactly the architecture PILOT names as the problem. Whether a supervisor detects the failures a worker misses is not measured in anything read — the benchmarks report task success, not false-completion rate.
Open Questions
- What model runs the supervisor? Nothing read says whether it is the same backbone, a smaller one, or a stronger one. The cost accounting turns on it, and so does whether "supervisor-worker" is a scaffold or a second model.
- The three benchmarks are not enumerated in the abstract beyond Terminal-Bench 2.0.
- "First in five of six configurations" leaves one. Which configuration, and by how much, is not stated.
- Terminal-Bench 2.0, not 2.1 or 3.0. This wiki holds figures on all three and they are not comparable across the version boundary — see GLM-5.3, which records that specific hazard.
- Live self-evolution writes to a persistent harness across runs. That is the cross-trajectory state FrontierChallenge: Evaluating Scientific Workflow Completion (arXiv:2608.24979)'s companion cluster is currently split on: it is the reported mechanism of Meta^n: Recursive Self-Improvement through Emergent Depth (arXiv:2608.24735) and the named failure mode of AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces (arXiv:2608.23041). PILOT does not cite either.
Cite
Xiao, Sun, Wu, Hui, Da, Luo, Chuan, Hu, Li, Jiang. PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents. arXiv:2608.26530, 2026. https://arxiv.org/abs/2608.26530
Related
- Eval Harness Configuration
- Agents (LLM Agents)
- JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution (arXiv:2608.25593)
- AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces (arXiv:2608.23041)
- Meta^n: Recursive Self-Improvement through Emergent Depth (arXiv:2608.24735)