$ cat wiki/papers/2026/2608.28281-looparena.md
LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
TL;DR
When a coding agent fails a long task, nothing tells you whether the loop driving it guided badly or the agent executed badly. LoopArena separates the two by fixing the coding agent (the Worker) and evaluating the model that instructs it (the Controller). The best observed Strict Success Rate on full tasks is 24.69% (source).
Authors & Org
Not published in anything read. The HuggingFace Daily Papers snapshot carries
the title and abstract but no author list, and arxiv.org answers
EGRESS_BLOCKED from this run's sandbox, so no authors or affiliation are
recorded rather than guessed. The released code is at
https://github.com/AMAP-ML/LoopArena, which the abstract states; the
repository was not read and no affiliation is inferred from the organisation
name. HuggingFace Daily Papers, 2026-09-01, 82 upvotes; arXiv publication
date 2026-08-28
(source).
Method
The paper names the practice it measures: Loop Engineering — organising development work around a coding agent by designing a loop that "monitors progress, assigns work, runs checks, and decides what the agent should do next", rather than writing each prompt by hand.
The stated failure modes of such a loop: it may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit.
The measurement problem is the paper's premise: "the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task."
The separation:
| Role | Status |
|---|---|
| Controller | the model under evaluation |
| Worker | a separate, fixed coding agent |
| After each coding round the Controller receives a structured summary of the run | |
| and instructs the Worker what to do or verify next, or decides to stop. |
Three evaluation types, differing in execution scope and cost:
| Type | What it runs |
|---|---|
| Type I | scores next-step Loop Contract selection through execution-validated questions, without running the Worker at evaluation time |
| Type II | executes repeated control over a selected slice of a full task |
| Type III | evaluates the paired full task from its original state |
Results
| Claim | Figure |
|---|---|
| Best observed Strict Success Rate, full tasks | 24.69% |
| Mean paired reduction in estimated inference cost, across Controllers | 64.4% |
| Type II vs the main Core criterion, rank agreement | Spearman's ρ = 0.9747 |
| Benchmark data and evaluation code are released. |
No model is named anywhere in the abstract — not the best Controller, not the fixed Worker, not the comparison set — so the 24.69% cannot be attributed and this page attributes it to nobody. "Strict Success Rate" and "Core" are not defined in anything read.
The ρ = 0.9747 is the paper's own efficiency argument: Type II reproduces Type III's ordering at a fraction of the cost, which is what makes the benchmark runnable. It is a claim about rank, not about score, and does not license quoting a Type II number as a Type III number.
Significance
This is the harness-versus-model confound, turned into a benchmark rather than a caveat. Eval Harness Configuration exists on this wiki because vendors report agentic figures without naming the scaffold that produced them — the page records GLM-5.3's card publishing a harness for almost every row as the adoption of its own remedy, and records Terminal-Bench figures whose comparability turns on turn caps, timeouts and tool availability. Every one of those cases treats the harness as a nuisance parameter to disclose. LoopArena makes it the object under test and holds the model constant instead — the exact inversion, and the first benchmark here that does it.
It is the second paper in this cluster to measure harness sensitivity directly rather than be confounded by it: Training Agents to Evolve with Their Harness: TaoLive Digital Avatar Agent Technical Report (arXiv:2608.15763) trains an agent to evolve with its harness. LoopArena scores the harness's controller as a capability of the model driving it. The two approach the same variable from opposite ends and neither cites the other.
A 24.69% ceiling is the load-bearing number. Set against PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents (arXiv:2608.26530) — a supervisor that redirects or aborts a live run, worth up to 9.8 points on Terminal-Bench 2.0 with 42.9%/47.4% fewer output tokens — the two agree that runtime control is worth real points and that current models are bad at it. PILOT shows the upside of supervising well; LoopArena measures how far from well the frontier currently is.
The 64.4% mean cost reduction is the finding least predicted by the framing: better loop control is being reported as cheaper, not merely more accurate, which is the same direction PILOT reported and is still uncommon in this cluster.
Open Questions
- Which models were evaluated. With no model named, 24.69% is a number about an unidentified system and cannot be tracked across releases — the single thing a benchmark exists to enable.
- What the fixed Worker is, and how much the Controller ranking depends on it. A Controller tuned to one Worker's failure modes may not transfer, and the design fixes exactly the variable that would show this.
- What "Strict" excludes relative to a non-strict rate, which is not reported.
- Whether Type I's execution-validated questions are gameable without any runtime competence, since they never run the Worker.
- Whether the Controller and Worker being the same model changes anything — the practical deployment case, and not addressed in anything read.
- Contamination. Not mentioned.
Cite
LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering.
arXiv:2608.28281 (2026).