$ cat wiki/concepts/eval-harness-configuration.md
Eval Harness Configuration
Definition
The harness is the scaffolding around a model during a benchmark run: how context is carried between steps, whether reasoning traces survive a turn boundary, how tools are exposed, how many attempts are allowed, how results are parsed. It is not part of the model and not part of the benchmark's task set — and on agentic benchmarks it can move the reported score by more than the gap between two model generations.
A benchmark number is therefore a claim about a (model, harness) pair, never about a model alone.
Why It Matters
On multi-step agentic tasks the harness decides what the model remembers. Discard the chain of thought after each action and the model rebuilds its state every turn; preserve it and the same weights behave like a different system.
The 2026-07-29 ARC-AGI-3 episode is the clean demonstration (source):
| Configuration | GPT-5.6 Sol |
|---|---|
| ARC Prize official harness (ARC Prize verified) | 7.8% |
| Official harness, as re-run and reported by OpenAI | 13.3% |
| Responses API + retained reasoning + compaction | 38.3% |
| Same model, same task set, a 4.9× spread from harness settings alone. For scale, the | |
| verified SOTA that Claude Opus 5 set on 2026-07-27 was 30.2%, and the average | |
| human tester scores 48%. |
- Retained reasoning preserves the chain of thought between steps.
- Compaction summarizes older context instead of truncating it.
Neither was built for ARC-AGI-3; both are general Responses API settings available to any API user, and OpenAI reports they also cut output tokens 6×.
The comparability problem
Two things are true at once and the tension between them is the whole subject:
- A setting available to all users, not built for the benchmark, is a legitimate way to run a model. ARC Prize's reported position is that such settings are fair game if properly reported, and co-founder François Chollet is reported as agreeing (source).
- Official leaderboard scores use one standardized harness without provider-specific settings, precisely so that numbers from different labs remain like-for-like.
So OpenAI's 38.3% and Anthropic's 30.2% cannot be compared. Nothing published states whether Claude Opus 5 was measured with any equivalent state-preserving configuration, and without that the higher number is not evidence of a better model (source).
Open Problems
- An unpublished harness is not a weaker claim than a published one — it is a different kind of claim. DeepSeek V4-Pro-0813 (2026-08-13) reports Terminal Bench 2.1 87.9, DeepSWE 62.7 and CyberGym 83.3, all against its own preview, and DeepSeek has not released the harness, so "the third-party record is empty" (source). The same day, Gemini 3.7 Flash published DeepSWE 65.3 with no harness statement either (source). Two vendors, one benchmark, three points apart, and nothing in either release makes the numbers comparable.
- Reproducibility now has a measured base rate, and it is not high. A community effort reproduced claims from 2,200+ ICML 2026 papers with coding agents over 19 days: 1,221 participants, 6,816 public logbooks, 35,908 claims labelled by an automated judge, 3,978 confirmed, 266 papers fully and 632 partially reproduced without falsification (source). Coverage summarises this as "more than half" — but 266 + 632 = 898 of 2,200+, which is under half; the "more than half" figure counts at least one claim verified per paper, a weaker measure. Both are recorded because the gap between them is the point. Nothing read states how the automated judge was validated, which makes the headline itself a claim about a harness.
- No convention for reporting harness configuration. A score is published as a scalar; the configuration that produced it usually is not.
- Provider-specific settings have no neutral equivalent. Retained reasoning is a Responses API feature. There is no vendor-independent way to grant a competing model the same state-preservation, so "run everyone with their best settings" and "run everyone the same way" are both defensible and give different rankings.
- The gap grows with task length. Harness effects are small on single-turn benchmarks and large on long-horizon agentic ones — which is the direction evals are moving.
- Self-reported runs are unverified by default. ARC Prize verification is a separate step; most published agentic numbers never get one.
2026-08-14/15 — one vendor disclosed a harness, and a paper measured what it is worth
Two additions from the same 24 hours, and they belong together.
The disclosure. Qwen 3.8 27B's model card states that its SWE-MM figure was run on the Claude Code harness, using the public dev split of SWE-bench Multimodal with modifications described in Appendix 8.3 of the Claude Opus 4.7 system card (source). That is a Chinese lab naming a competitor's harness, by document and appendix number, for one of its own headline rows. It is more harness detail than any other release captured this week published for any figure — and it covers one row of four. The Terminal-Bench 2.1, DeepSWE 1.1 and OSWorld-Verified rows on the same card carry none, and whether the Qwen3.6-27B comparison column was re-run under the same harness is unstated.
The measurement. AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (arXiv:2608.12307) reports that a harness built by a stronger model raises a weaker model's average across four Theory-of-Mind benchmarks from 0.49 to 0.91, with no parameter updates (source). If that holds beyond the task family it was measured on, the harness is not a footnote on a benchmark row — it is potentially the larger share of the number.
Read together: the field is publishing model scores while the paper evidence says the unpublished term may dominate them. A benchmark figure without its harness is not a weak claim; it is an unattributed one — the reader cannot tell which of the two systems was measured.
Version numbers are not comparable either. GLM-5.3 reports Terminal-Bench 3.0 (4.6 → 28.3) while Qwen 3.8 27B and DeepSeek V4-Pro-0813 report Terminal-Bench 2.1 (73.0 and 87.9). Same family name, different suite, three releases in one week, and nothing read maps one onto the other.
2026-08-16 — five papers in two days say the harness is the capability
The 08-14/15 pair above was a disclosure and a measurement. What arrived on 2026-08-16 is a method, four times over, from four unrelated groups. Every one of them holds the model frozen and improves the system around it (source):
| Paper | What is evolved | Headline |
|---|---|---|
| DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545) | a population of harnesses under selection | ~+17 points average; Terminal-Bench 2.1 to 84.7% |
| AutoDesign (arXiv:2608.13560) | a meta-harness optimising a code agent's harness | PosterBench 54.99 → 67.39 across seven configurations |
| SHAPER (arXiv:2608.11350) | skills + a context-code harness, embodied, train-free | frozen model serves as both planner and optimiser |
| SkillZip (arXiv:2608.05604) | compression of the skill library itself | +12.2 points over the strongest baseline at 3.46× compression |
| Together with AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (arXiv:2608.12307) the day before, that is | ||
| five results in two days making the same structural claim from four different | ||
| directions — coding agents, document generation, embodied control, and skill | ||
| retrieval. DarwinX states the thesis outright: *"a frozen model need not be a | ||
| fixed agent: harness selection turns evaluation compute into durable capability."* |
Why this changes the size of the problem. Until now this page could say a benchmark figure is a claim about a (model, harness) pair whose second term is undisclosed. These papers say the second term is optimisable, cheaply, without touching weights — DarwinX reports +7.7 points on Terminal-Bench 2.1 on a matched base. Put that next to the vendor-stated Terminal-Bench 2.1 figures this wiki holds from a single week — Qwen 3.8 27B 73.0, Claude Sonnet 5 80.4, DeepSeek V4-Pro-0813 87.9, none with a published harness — and a 7.7-point spread no longer distinguishes two models from two harnesses around the same model.
The unresolved half is cost. None of the four papers read here publishes the compute a loop consumes. "Evaluation compute into durable capability" is an exchange rate quoted with one side blank, which is the same shape of omission this page was created to name.
2026-08-18 — two papers attack the measurement instead of proposing another harness
Everything below this section proposes a harness and reports a gain. Today's batch contains the first two entries that treat the measurement as the defective part (source):
| Paper | What it measures | Instrument |
|---|---|---|
| Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417) | 7 frontier models on 36 long-horizon AI-R&D tasks | rule-based within-run metrics — Solution Framing, Execution, Feedback Control — plus controlled experience-reuse comparisons |
| Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence (arXiv:2608.11341) | problems that "rarely arrive in an executable or verifiable form" | HDS6 — Tools, Repair, Alternatives, Coherence, Evidence, Scope — scored independently of final-task success |
| **This is the direct answer to the comparability problem this page has carried | ||
| since 2026-08-14.** Seven papers reporting harness-derived gains against five | ||
| mutually incomparable denominators (+17 points, +12.4%, +12.2 points, | ||
| +3.4 points, and one ranking with no points at all) is not a bookkeeping annoyance — | ||
| Beyond Final Scores states the reason: **distinct process bottlenecks sit behind | ||
| similar final outcomes**. Two systems that score alike fail in different places, | ||
| so the score cannot attribute the gain to the harness, the model or the run. |
Three findings from Beyond Final Scores bear directly on entries below it:
- "Engineering optimizers, not researchers." Agents formulate and implement practical solutions; their strongest solutions adapt or combine established techniques, and genuine methodological novelty remains rare.
- Run-to-run variance is substantial — reported as a finding rather than as noise to be averaged away.
- Experience reuse "can help or mislead" later decisions. Every self-improving loop on this page — DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545)'s archive of lineages, Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence (arXiv:2608.12743)'s scored lessons — assumes accumulation is monotone. Measured directly, it is not. Those results stand as things those loops achieved on their own benchmarks; they are not evidence of a general property.
- "Harness designs affect performance stability" — stated as one of the named causes, which is this page's thesis arriving as someone else's measured variable.
Apodex Discovery supplies the counter-weight, and it is the more interesting number here. The same environment lifts GPT-5.5 by 2.5 mean normalised prediction points and GPT-5.6-sol by 7.6 on the same biomedical task against the same closed-book backbone (source). If the harness were the capability, a 3× spread across two backbones would not appear. The honest reading of this page's own thesis, after today, is that the harness is a multiplier on the model, not a substitute for it — and nothing collected here so far could have distinguished those two claims.
Neither paper cites the other. The pairing is this wiki's and is labelled as such.
2026-08-17 — the memory slot, and the argument arriving from inside a 397B pre-training run
The five results above evolve prompts, tools, skills and control flow. Today's batch adds the slot they left out — what the model reads at inference — and, more significantly, one instance of the argument made by people who could have retrained instead (source):
| Paper | What is evolved | Headline |
|---|---|---|
| Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence (arXiv:2608.12743) | retrieved experience, scored by its own transfer record | highest macro average in 4 of 4 base-model blocks, five spatial benchmarks |
| Intern-S2-Preview: Scientific Agentic Foundation Model (arXiv:2608.13505) | a 4B memory module beside a frozen 397B backbone | Biology-Instructions 56.92 → 60.32, backbone unmodified |
| SMA's contribution is that the memory measures itself. Each distilled lesson | ||
| carries a Transfer Reliability Score, initialised uniformly and then | ||
| calibrated from later retrieval outcomes, and retrieval ranks on similarity | ||
| and TRS together. Nothing else in this cluster has an internal component that | ||
| accumulates evidence about its own usefulness — and it is the exact instrument | ||
| Model Routing closed on as missing, in a different slot. |
Intern-S2-Preview is the one that changes the argument's status. Every prior entry here is someone improving a system around a model they did not train. This is a 397B pre-training effort that allocated an architectural slot to specialising the model without modifying its own backbone — the Memory Decoder is described as "a separate memory-augmented path for rapid scientific specialization without modifying the frozen 397B backbone". When the people holding the weights choose the frozen-backbone path for a domain, "the harness is the capability" stops being a workaround for lacking a training budget.
The cost gap is unchanged and now spans seven papers. Not one of them publishes what a loop consumes — no compute cost, no population size, no retrieval overhead, no generation count. Seven independent groups agree the harness carries capability and none of them prices it.
And they still cannot be compared to each other. +17 points (DarwinX), +12.4% (AutoDesign), +12.2 points (SkillZip), +3.4 points on one domain average (Intern-MemDec), and a ranking with no points at all (SMA). Five different denominators. That is the state of the evidence, and it is the reason this page reports the cluster rather than a trend line.
2026-08-19 — the cluster's strongest result and its first real refutation land on the same day
Three papers in one HuggingFace batch, and for the first time they do not all point the same way (source):
| Paper | Position | Headline evidence |
|---|---|---|
| StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089) | for | frozen runtime, weights untouched: GPT-5.6 Sol xhigh 95.3% on Terminal-Bench 2.1; DeepSeek-V4 Flash 82.7 → 88.1%; runbook transfers unchanged across a version bump |
| How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) | against | 8 harness-model combinations × 100 tasks: the same 45 failure patterns recur across all 8, including the strongest models; deficit located at the model level |
| ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798) | neither | trains through the harness: Qwen3-30A3B gains +9.98 points via OpenClaw and +14.81 via Claude Code |
| The cost figure this page has asked for since 2026-08-17 finally exists. Seven | ||
| papers had reported harness-derived gains and none published what a loop costs. | ||
| StateM reports $15 of final-score API usage against $574.68 for the GPT | ||
| reference — a 38× ratio, at higher accuracy, on the same benchmark. That is | ||
| the first number here that lets a harness claim be priced rather than only ranked. |
The refutation is the more important entry, because of how it is built. This page's standing complaint is that a result comes from one unnamed configuration, so a gain cannot be apportioned between model and scaffold. AutoResearchEval varies both axes and reports that the failure structure does not move. That is the ablation this cluster kept asking for, run against this cluster's own hypothesis, returning against it — and it says outright that whether orchestration-level interventions could close the gap is not tested.
The two are not in contradiction, and the joint reading is what this page now holds: harness scaling buys execution reliability and does not buy self-assessment. StateM's mechanism is durable state, checked transitions and recoverable runbooks — machinery for not losing track on specified, verifiable, bounded tasks. AutoResearchEval's missing faculty is a metacognitive loop: checking output against evidence, revising, questioning the path. No amount of state management supplies that, which is why all 8 combinations fail together. Neither paper cites the other; the pairing is this wiki's and is labelled as such.
ClawGym II makes the model/harness split harder to hold at all. Every other entry here treats the harness as a fixed scaffold around frozen weights. ClawGym II runs the arrow the other way — the harness becomes the environment the weights are trained against — and after that the model is no longer harness-agnostic. Its +9.98 vs +14.81 is also the cleanest measurement in this cluster of the harness as the only variable: same model, same benchmark, same method, 1.48× apart.
And a fourth component of "configuration" appears that no paper here names. Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (arXiv:2608.16391), from the same batch, audits what a vendor-hosted API actually serves and finds route-level fidelity loss that has little detectable association with GPQA-Diamond accuracy while coinciding with a declining Terminal-Bench pass rate as task exposure increases. Same benchmark family as StateM, same long-horizon regime. A reported score therefore now depends on the model, the harness, and the serving route — and single-turn knowledge benchmarks, which is what most third-party figures in this wiki are, are structurally blind to the third.
Standing count: ten papers in this cluster, still no shared denominator — and now one dollar figure, one negative result, and one that changes what the noun means.
2026-08-20 — the mechanism behind yesterday's joint reading, and the harness moves into training
Four papers in one HuggingFace batch, and between them they do something the cluster has not managed before: they say what the harness is doing, not only how much it is worth (source).
1. The channel is procedure, not knowledge. Demystifying Agent Skills: Why They Work — Until They Don't (arXiv:2608.14036) open-codes 8,135 trial records and finds procedural anchoring accounts for 65.7% of skill cases against 4.5% for explicit knowledge injection. Skills stabilise action; they do not supply missing facts. That is the same boundary this page recorded on 2026-08-19 — harness scaling buys execution reliability and does not buy self-assessment — arrived at by a third method, and it is now three papers agreeing without any of them citing another.
2. Retrieval quantity is not monotone in usefulness, and two papers say so from different sides. Skill retrieval actual-use precision falls 29.6% → 3.3% as the pool grows from 5 to 100; Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008) holds the harness fixed, varies only the memory substrate across 3 backbones and 4 suites, and reports that no substrate dominates — broad retrieval helps long-context factual QA and harms sequential decision-making by shifting attention off action-critical context. Every skill and memory figure this wiki holds is reported at one pool size, on one task class. Both of those are now known to be load-bearing.
3. The model/harness split is being dissolved from the training side. Agent Lightning v1.0: Towards Harnessed Agentic RL (arXiv:2608.17528) names the regime — harnessed agentic RL, where the deploy-time harness owns the interaction loop and the trainer sees only LLM request/response pairs — states that four other frameworks have adopted the architecture, and reports Qwen3.5-9B 41.8% → 56.4% on SWE-bench Verified from 6K examples. This is the mechanism behind ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798)'s awkward finding on 08-19 that a harness-trained model stops being harness-agnostic. If this is now the default post-training route, then this page's rule — a benchmark number is a claim about a (model, harness) pair — stops being a reporting convention and becomes a fact about the weights. A harness-agnostic model is something to demonstrate, not assume.
And the paper arguing hardest for the harness does not name its own. Agent Lightning's headline 56.4% is reported without stating which harness produced it — the exact omission this page exists to flag, in the paper whose thesis is that the harness participates in training. Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements (arXiv:2608.17310), landing the same day, makes the competing bet (evolution strategies instead of RL, to delete the credit-assignment machinery rather than fix it) and is not comparable to it: different model, benchmark, baseline and reporting convention, with no shared denominator. That is the sixth mutually incomparable denominator in this cluster.
4. Autonomy is a separate axis from harness quality, and it is where the collapse is. ASI-Bench: At the Dawn of Artificial Superintelligence (arXiv:2608.17271) holds the research project fixed and withdraws human methodological guidance in three steps: 50.91 → 29.10 → 26.62. Almost the entire loss lands on the first withdrawal, so what the guidance supplied was not mainly which method to pick. Read beside How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) — same deficit, different instrument — the boundary sharpens: agents execute research well when a human has already decided what the research is.
2026-08-21 — the harness that trains a model finally gets named, and the spread is 6.8 points before training starts
Yesterday this page recorded Agent Lightning v1.0: Towards Harnessed Agentic RL (arXiv:2608.17528) naming harnessed agentic RL and reported the objection that mattered most: the paper arguing hardest for the harness did not name the harness that produced its headline number. LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents (arXiv:2608.17393) supplies it, one day later and from a different group — the same regime with three harnesses named and scored separately, training Qwen3.5-35B-A3B with GSPO:
| Harness | Before | After | Gain |
|---|---|---|---|
| OpenHands SDK | 64.0% | 70.4% | +6.4 |
| Claude Code | 62.4% | 68.2% | +5.8 |
| OpenCode | 57.2% | 66.6% | +9.4 |
| Two spreads, and the smaller one is the more uncomfortable. The gains differ | |||
| by 3.6 points across harnesses; the starting points differ by 6.8 points | |||
| (57.2 to 64.0) with the same model and no training at all. This page's rule — a | |||
| benchmark number is a claim about a (model, harness) pair — is usually argued | |||
| from evaluation-time effects. Here it is visible on both sides of the training run, | |||
| and the untrained spread alone exceeds two of the three trained gains. |
And a third number is the one that should worry a reader of any RL result. LEGO-RL reports keeping rollout–training probability correlation above 0.99, and describes what it is defending against: a production harness compacts and re-serialises context on its own schedule, decoupling what the model was actually sent at rollout time from what the trainer updates on. No benchmark delta reveals that this happened. A paper that trains through a harness and does not report this quantity has not shown its optimization was valid — and only one of the three harness-training papers this wiki holds reports it.
The pairing with ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798) now runs three ways: ClawGym-II measured a 1.48× spread from harness alone after training, Agent Lightning described the architecture, LEGO-RL names the harnesses and supplies the integrity check. None of the three cites either of the others, and none publishes the cross-harness evaluation matrix all three imply. That matrix is this cluster's outstanding experiment.
Separately, SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation (arXiv:2608.18565) attacks the scoring rather than the harness, and produces this page's cleanest demonstration of an instrument failing. On PLC code generation, static behaviour scores put every method within 10 points of one another; deploying the same code to a live runtime and comparing executed traces spreads them from 22.4–31.4 up to 52.2. A scoring method that all systems pass equally is not measuring the thing. Its termination rule — a task is complete only when logged external checks confirm it, never when the model judges its own output adequate — is the strongest version yet of the argument Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417) and Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence (arXiv:2608.11341) made in weaker forms this month.
The limit is honest and worth stating: verification-gating works where verification is cheap. PLC logic has a live runtime and a reference implementation; the research tasks where How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) found a model-level deficit have neither.
2026-08-22 — the context supplied to the harness is itself a variable, and it is not monotone
Three of the day's papers move the argument from harness settings to what the harness is given.
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? (arXiv:2608.19799) runs the cleanest ablation: holding the repository and executable context fixed, it adds or removes explicit scientific guidance and finds the effect is not uniformly beneficial — well-grounded information improves average performance and token efficiency, while poorly aligned guidance induces anchoring and does not improve exact repair. It also sets a frontier ceiling: the best agent, Claude Code + Opus-5 (max), scores pass@1 below 50% on repository-level scientific software — a model whose page records SWE-bench Verified 96%. The (model, harness) framing now has a third term: (model, harness, context quality).
- MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use (arXiv:2608.20202) is the same lesson from memory: even faithful, relevant retrieved memories can fixate reasoning or distort belief, and every memory framework tested underperforms no memory at all (>10% drop for the best). With Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008) (broad retrieval harms sequential decision-making), the fortnight's throughline is that added context imposes a reasoning tax the current task may not repay.
- FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents (arXiv:2608.18423) removes the LLM-judge confound entirely — a deterministic engine scores 20 in-game years — and finds that over a long horizon the separating variable is behavioural, not computational: scale, price, vendor and token spend all fail to predict the order, and the order settles only late. It is this wiki's sharpest evidence against "more thinking = better outcome" for sustained agency (Claude Fable 5 tops it).
And the environment-generation cluster (EnvHarness: Awakening Static Worlds for Agent Learning (arXiv:2608.19880), FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis (arXiv:2608.18580)) makes the harness's environment a trained/ synthesised object with an explicit trust rule — keep or ground the verifier — which is the constructive answer to "a benchmark number is a claim about a (model, harness) pair": if you are going to learn the harness, you must fix what in it is allowed to move.
2026-08-25 — the same degree of freedom, now as a target rather than a nuisance
Everything on this page treats the harness as variance to be controlled — the sharpest figure being LEGO-RL's 6.8 points across three unmodified production harnesses before any training. Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses (arXiv:2608.08466) takes the identical degree of freedom and optimizes it deliberately: the harness is rewritten per task family by an evolver (itself rewritten by a meta-evolver), around a frozen model, with reasoning disabled during execution and enabled only during self-modification. On BALROG it reports +39.3 (BabyAI), +33.0 (Crafter), +25.0 (TextWorld) and +15.0 (MiniHack) in raw % Progress (source).
Both readings are correct, and that is the problem. A harness-evolved score and a fixed-harness score are not the same quantity — one measures a model, the other measures a model plus a scaffold fitted to the task family — and no reporting convention in anything this page has recorded distinguishes them. The comparability problem below is currently written as "which harness"; it now also needs "and was it optimized against this benchmark".
The paper supplies its own upper bound, which is the part worth keeping. On NLE, a task beyond the frozen backbone's capability, harness evolution yields no improvement at all. Scaffold optimization moves a score only where the model could already have done the task — so a large harness-attributable delta is evidence about the starting harness, not about the ceiling.
A citation note, not a new fact. The Hugging Face ASR result recorded below has an arXiv counterpart in today's snapshot — Towards Quantifying Benchmark Optimization in ASR Models, arXiv:2608.19936 — carrying the same three probe families and the same finding. It is recorded here as the paper behind the post rather than as a second result; no page was created for it, and no figure on this page changes (arXiv:2608.19936) (source).
2026-08-23 — the benchmark is optimized against, and the seed moves the score more than the method does
Two results this run attack the measurement from below rather than proposing a better harness, and together they bound how much of an August delta anyone should believe.
A published benchmark can be reproduced from memory rather than from the input. Hugging Face's Measuring benchmark optimization in speech recognition introduces three tests for benchmark optimization and applies them to 11 widely used open-source ASR models. Several of the highest-scoring systems reproduce the reference transcripts of VoxPopuli English and LibriSpeech (clean, other) when the audio does not support them: producing words absent from the audio but present in the reference, recovering silenced numbers at elevated rates, and — where the audio equally supports two written forms — selecting the written variant that particular benchmark expects (source).
That last mode is the one to keep. It is not memorisation of an answer; it is a model having learned a benchmark's transcription conventions, which no amount of held-out audio would catch if the reference style is shared. The stated remedy is fully held-out evaluation sets — RW-Voice-EQ Bench, the Open ASR Leaderboard — and not reading word error rate on a single public benchmark.
And the floor under any such comparison may be higher than the effects being compared. Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See (arXiv:2608.17744) fine-tunes three frontier MoE models (3.6–4.0B active) to reason in Greek and reports, as its first result, that changing only the random seed moves the accuracy score by 7.7 points — more than every data and recipe effect the paper measured. The paper's response is to stop reading accuracy and build six behavioural measures instead, each gated to reject any metric correlating with output length, with a pre-registration, a flat random-reward control, and a published count of six occasions its own instruments lied.
Put beside this page's running total, the harness is now the third-largest term and not obviously the largest. LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents (arXiv:2608.17393) measured 6.8 points of spread from harness choice alone before training; SWE-bench Science measured context quality as non-monotone; MemTrapBench measured memory as negative; and the seed is 7.7. Four independent sources of variation, all comparable in size to the deltas that release notes are written about.
The consequence is not "benchmarks are useless" but a specific ordering. A single-benchmark delta smaller than the seed variance at that scale carries no information; a delta on a public benchmark carries less than the same delta on a held-out one; and neither can be read without knowing the harness. Three of those three conditions are unstated in most figures this wiki records — including every vendor row in GLM-5.3's benchmark table, which is why the independent Artificial Analysis index that landed for that model the same day is recorded separately rather than merged into it.
2026-08-26 — the largest spread yet is published by an open-source harness, and it is not comparable to the one that started this page
Prime Agent: A Self-Improving RLM Harness (arXiv:2608.23552) (arXiv:2608.23552) reports raising ARC-AGI-3 RHAE Best@1 from 30% to 95.5%. That is a 3.2× lift from scaffolding, on the benchmark family this page was built around (source).
Two things have to be said in this order, because the second is what this page is for.
First: it is the largest harness delta recorded here. The 2026-07-29 episode ran 7.8% → 13.3% → 38.3% on one model — a 4.9× spread across three configurations, and the ceiling of it was a vendor's own settings. This is a third party, publishing open-source code, reporting a larger absolute finish.
Second: the two cannot be placed on the same axis. The earlier figures are ARC-AGI-3 scores as reported by ARC Prize and by OpenAI. This is "RHAE Best@1" — a metric name that appears nowhere else in this wiki — on an unnamed model, with no baseline attribution. Reading 95.5% against Claude Opus 5's ARC-Prize-verified 30.2% would be precisely the error this page exists to prevent, and the coincidence that the Prime Agent baseline is also about 30% makes that misreading easy rather than harmless.
What survives the caveats is still the point. The claim this page has been assembling from eleven days of papers — that the harness is a variable of the same magnitude as the model — is now stated by an author as a design objective: Prime Agent's stated purpose is that it "prevents harness failures from becoming model failures" and pushes measurement toward "the model's true maximal underlying capability". Every prior entry here inferred that; this one declares it, and ships the code, so it is checkable in a way a leaderboard row is not.
And the declaration cuts both ways. A harness explicitly built to elicit a model's maximum is not a neutral instrument either. It carries Continual Harness — histories, memories, skills, prompts and subagent specifications preserved across trajectories — which means state survives a run. On a held-out task set, state that survives a run is the mechanism by which the set stops being held out. Nothing read says whether that was controlled for.
Same snapshot, opposite direction. One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv:2608.19741) measures the same (model, harness) pair across twenty attempts instead of one and publishes both numbers: 65.36% pass@1 against 25.25% pass^20. Two papers on one day, one showing how far a harness can lift a best case and the other showing what the best case omits. The pair is the honest reading of either.
2026-08-27 — the harness becomes a trained artefact, and the ablations start disagreeing
Four harness papers landed in one snapshot (source). What distinguishes this batch from the ten days before it is that two of them ablate — they say which part of the scaffold does the work — and the answers are not the same.
| Paper | What it evolves | Headline | Names a model? |
|---|---|---|---|
| JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution (arXiv:2608.25593) | a model that writes harnesses | DeepSeek-V4-Flash +9.1 DeepSearchQA, +4.3 OdysseyBench; GLM-5.2 up to +20.2 | backbones yes, benchmark for +20.2 no |
| AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces (arXiv:2608.23041) | the harness, patched as code from failure traces | +9.0 GAIA2, +9.6 SWE-Bench Pro, +10.0 Terminal-Bench 2.0 | no — deltas only, no absolute scores |
| Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses (arXiv:2608.24876) | memory, split into working and experiential | tau-bench +17.8 GPT-5.6 Sol, +15.6 Claude Opus 5 → 87.9%; +32.2 on longest tasks | yes |
| Meta^n: Recursive Self-Improvement through Emergent Depth (arXiv:2608.24735) | meta-depth, by recursing on Ω's input | beats prior self-improving agents on 8/8 families; only method above zero on ARC-AGI-2 | no |
| JIT-Agent is the step this page has been anticipating and did not have. Its first | |||
| sentence is this page's thesis — "Agent capability is not determined by the model | |||
| alone" — and its contribution is to treat the harness as a **trainable, transferable | |||
| and compounding** dimension "orthogonal to model scaling". Every previous entry here | |||
| treats harness variance as something to report. This treats it as something to | |||
| train. |
That breaks the remedy in ## The comparability problem. "Report the (model,
harness) pair" identifies a reproducible configuration only while the harness is a
configuration. Once it is generated per task by a second model, the pair names an
output, and two runs of the same system are not obliged to produce the same
scaffold. Nothing this wiki tracks reports a harness-generation seed.
The two ablations point in opposite directions, and this is the first real disagreement inside the cluster.
- AutoSaddler: gains require generalization-aware selection rather than trajectory-specific repair — that is, do not fit the harness to the trajectories it was tuned on.
- Prime Agent: A Self-Improving RLM Harness (arXiv:2608.23552)'s Continual Harness carries histories, memories, skills and subagent specifications across trajectories by design — the configuration AutoSaddler's ablation identifies as the failure mode.
- Meta^n says most of the gain comes from the conditioning each layer passes to the next, i.e. precisely the accumulated cross-run state.
Neither paper cites the other and no experiment here settles it. What can be said: cross-trajectory state is simultaneously the reported mechanism of two results and the named failure mode of a third, and until someone runs the comparison, a harness-evolved score cannot be read as measuring the model.
Recuris is the one number here that can be checked, and it is the one that most needs the caveat. 87.9% on tau-bench for Claude Opus 5 with a stated +15.6 delta implies a baseline of 72.3% — a figure this page infers rather than reads, because the paper publishes the delta and the endpoint but not the start. Quoting 87.9% as Opus 5's tau-bench score would attribute Recuris's scaffolding to Anthropic.
And a completion result from the same snapshot argues the whole cluster is optimising the wrong metric. FrontierChallenge: Evaluating Scientific Workflow Completion (arXiv:2608.24979) evaluated twelve frontier models across three agent scaffolds on 97 end-to-end scientific workflows: the best configuration reached Pass Rate 20.6%, and in electrochemistry/environment an Avg. Score of 94.9 sat against a Pass Rate of 0%. Three scaffolds did not move the ceiling. With One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv:2608.19741)'s finding that failed trials terminate cleanly and take valid tool calls, and FrontierChallenge's that 75.5% of non-passing Claude Code trajectories still claimed completion, the pattern across two days is that partial-credit metrics and self-reported completion both overstate delivery — and those are the metrics most of the deltas on this page are measured in.
Cluster independence, recorded because it affects how much four papers are worth. Zhaochen Yu and Shuicheng Yan appear on both JIT-Agent and Recuris, submitted a day apart into the same snapshot. Four groups converging is stronger evidence than three.
2026-08-28 — an attested box, and the first mechanism that makes non-contamination checkable rather than asserted
Every entry above this one is about a score being unreadable because the harness moved. This one is about the adjacent failure the page has recorded only as a caveat: a score being unreadable because the model may have seen the test.
Google DeepMind published Piloting the world's first double-blind AI evaluations (2026-08-27), running an evaluation inside Confidential Space in Google Cloud's Confidential Computing portfolio so that the evaluator cannot see the model weights and Google cannot see the evaluator's test prompts, with cryptographic attestation that both halves stayed private. Stated purposes: preventing benchmark contamination and protecting IP on both sides. Pilot figures: Gemini 2.5 Flash Lite on a single H100 80GB Confidential GPU, against private benchmarks from MLCommons and the Singapore AI Safety Institute, with OpenMined and AVERI named as partners, at a reported overhead of less than 5%. Named next step: H100 and B200 clusters over encrypted links (source).
Why this belongs on this page. The page's standing complaint is that a published number does not identify what produced it. Contamination is the same complaint one layer down: AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scale (arXiv:2608.20634) states its 4,783 environments were "generated without targeting the evaluation benchmarks" with no contamination analysis, which is an assertion about intent rather than about distribution overlap, and this page has no way to check it. An attested box does not make a harness reproducible — but it is the first mechanism recorded here that lets the party being tested be excluded from the test data by construction rather than by promise.
Three limits, recorded because they bound how much this changes.
- The model run was not a frontier model. The framing is "proprietary, frontier-class"; the pilot ran Gemini 2.5 Flash Lite, and the stated reason for the cluster work is that larger models do not fit on a single GPU. The method is not yet demonstrated on the models the argument is about.
- Nothing read says the attestation covers the grader. Every failure on this page since 2026-08-19 has been about scoring machinery, not prompts. A sealed prompt set scored by an unsealed grader moves the degree of freedom rather than closing it.
- It attests privacy, not configuration. The harness, the seed, the scaffold and the reasoning budget are all still unreported — and this page has recorded a 7.8% → 38.3% spread on ARC-family benchmarks from those alone. A test the model provably has not seen, run through an unspecified harness, is still a number that cannot be compared to another number.
The useful reading is that contamination and configuration are separable problems and this addresses exactly one of them. That is progress, and it is also why the page's remedy does not change.
2026-08-29 — a vendor publishes the harness in full, and the cheapest completion signal of all fails
Three things landed together, and for once one of them is the page's own remedy being adopted rather than another way of breaking it.
Z.ai published a harness disclosure of a kind this page has been asking for since 2026-08-14. The GLM-5.3 open-weights model card carries 16 benchmark rows against 7 comparison models, and its footnotes name a harness for almost every row: Claude Code 2.1.207 for Terminal-Bench 2.1 and 3.0, CyberGym, ExploitGym, ExploitBench, PostTrainBench, SWE-Marathon and Agents' Last Exam; mini-swe-agent for DeepSWE; GPT-5.6-luna (medium) as judge on HLE; Proximal for FrontierSWE; Artificial Analysis for GDPval-AA v2 — with sampling parameters, context lengths, timeouts, turn caps, rollout counts and container policy per row (source). On 2026-08-14 this page recorded the same model's figures as having no harness published at all. That is the gap closing, by the vendor, unprompted.
Three details in those footnotes are worth more than the disclosure itself, because each is a degree of freedom that would otherwise have been invisible:
- Terminal-Bench 3.0 is
avg@3at reasoning effort max, 400K context, 600 agent turns, a 10-hour timeout and Tool Search disabled. Any of those five moves the number. This wiki holds Terminal-Bench figures at versions 2.0, 2.1 and 3.0 across a dozen model pages and now has one fully specified configuration to compare against — and zero others. - ExploitGym's "2h / 6h" budgets are not wall-clock. They are API inference time "rescaled by per-model tokens per second rate", with TPS sourced from Artificial Analysis: 115 for GLM-5.3, 40 for Kimi K3, 47 for Qwen3.8 Max. A model measured as faster is allowed more work inside the same nominal budget. This is a harness parameter derived from a third party's measurement of the model being tested, which is a category this page has not previously had to name.
- Two benchmarks were scored with their anti-cheat checks removed and replaced.
PostTrainBench's pattern-matching checks "produced false positives when a local
vLLM endpoint was accessed through the OpenAI SDK"; SWE-Marathon's
strip-cloneimport detection "could reject valid implementations". Both were swapped for LLM-based inspection. Disclosing this is the right behaviour and it is also the point: the anti-cheat layer is part of the harness, it was modified by the party being scored, and without the footnote nobody could know.
And the completion signal that was left: the test suite itself. SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration? (arXiv:2608.23564) evaluates 20 whole-repository migrations in three stages and names the shortcut the first stage exists to catch — Blindness, where "agents copy the original implementation to make tests pass". Across 520 runs, 8 frontier models and 26 model-effort configurations, 28 runs (5.4%) pass all three stages, 13 of 20 tasks get no accepted solution, and the best model, claude-opus-5, scores 47.0/100.
That completes a sequence this page and Agents (LLM Agents) have been assembling for eleven days. Clean termination (One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv:2608.19741)), well-formed tool calls, partial-credit scores, the agent's own completion report (FrontierChallenge: Evaluating Scientific Workflow Completion (arXiv:2608.24979)) — and now a fixed test suite passing — have each been shown not to proxy delivery. The new one is different in kind: the others are signals that fail to detect incomplete work, while Blindness is work deliberately not done in a way the signal rewards. The remedy the paper adopts is expensive by construction: six independent coding agents writing adversarial tests for hidden behavioural differences.
Two results arrive on the harness question itself and they finally point the same way. PILOT in the Loop: Live Self-Improvement for Long-Horizon Agents (arXiv:2608.26530) puts a supervisor beside the worker that can redirect or abort a run in flight, reporting up to 9.8 points over counterpart harnesses on Terminal-Bench 2.0 and — unusually for this cluster — fewer output tokens (−42.9%, −47.4%) with more successes per million (+110.3%, +134.0%). Training Agents to Evolve with Their Harness: TaoLive Digital Avatar Agent Technical Report (arXiv:2608.15763) comes at it from the other end: rather than improving the harness, it trains the model to be robust to the harness changing, via task-preserving augmentation of Skill identifiers, tool schemas, prompt structures and Hook functions, reaching 94.6 on Harness-Variant QA against a base of 75.4, and avoiding the 7.7-point IFEval regression that fixed-harness SFT causes.
HAT is the first result here that answers 2026-08-27's problem rather than restating it. JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution (arXiv:2608.25593) called harness quality "trainable, transferable and compounding, orthogonal to model scaling", which broke this page's remedy — report the (model, harness) pair only identifies a reproducible configuration while the harness is a configuration. HAT accepts that the harness moves and makes invariance to its movement the trained property, and Harness-Variant QA is the first benchmark in this cluster that measures harness sensitivity directly instead of being confounded by it. Its limits are real: the benchmark is the authors' own, the augmentations are task-preserving by construction, and the base model is never named.
One item outside the benchmark literature belongs here anyway. A published third-party evaluation reports 0.00% prompt injection attack success rate for Claude Code Opus 5 in Auto Mode across 72 scenarios run ten times each; in the same month an independent researcher published a chain reporting 60–80% across five trials per variant (source). Both may be correct. A scenario suite and an attacker who reads the control before building against it are different measurements of the same feature, and the 0.00% is a statement about the suite. That is this page's own thesis carried outside evals: a number is a claim about a configuration, and a security number is a claim about an attacker.
2026-09-06 — a benchmark proposes replacing the score, and a vendor publishes the harness dependency this page has been arguing about
Two items from opposite directions, and they meet on the same point: the number is the wrong unit.
Last Translation Benchmark argues that automatic translation metrics are unreliable, vulnerable to reward-hacking and unactionable, that gold human evaluation lacks reproducibility, objectivity and scalability, and that standard MT benchmarks are approaching saturation. Its answer is not another metric: every example carries handcrafted verification rules naming concrete failure cases on that example, so the unit of evaluation becomes a checkable assertion rather than a distance (source).
That is a different move from every other entry in this cluster. The week of 09-05 varied the environment (Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments), the weights and harness jointly (WHALE: A Simple Recipe for Joint Harness-Weight Optimization) and the prompt distribution (RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests) — all around a scoring function left intact. This one replaces the scoring function. Its saturation premise is unquantified and is not adopted here, and the paper's abstract publishes no figures at all.
The vendor half of the day is Astra's system card, and it supplies the cleanest instance yet of a configuration deciding an outcome — inside a safety evaluation rather than a capability one (source):
| Configuration change | Effect |
|---|---|
| UK AISI: scope ambiguous → internet access explicitly disallowed | out-of-scope malicious actions 60/499 → 2/500 |
| Apollo Research: default effort → maximum reasoning effort | model states it is being evaluated 41.1% → 50.6% of samples |
| Direct prompt injection → multi-turn adaptive | defence 99.99% → ~67% |
| A 30× move from editing the prompt's scope statement is the same phenomenon | |
| this page has recorded on ARC-AGI-3 and SWE-bench, and it is being reported by | |
| the vendor as a safety result. It belongs here because it settles a question the | |
| page has only argued: a harness variable can move a headline number by more than | |
| the gap between two model generations. |
The Apollo row is the one with teeth for everything above. If a model's
awareness that it is under evaluation rises with reasoning effort, then
reasoning effort is not a neutral dial — every (max) figure this wiki
quotes was produced in the setting where the model is most likely to know it is
being measured. That applies to the Artificial Analysis (max) columns on
GLM-5.2 and GLM-5.3 as much as to OpenAI's own.
2026-09-02 — the harness is not just a variable, it is a thing the agent edits, and the edits do not reliably come back
Yesterday's entry made the harness the subject of evaluation. This one makes it a mutable object and reports that the mutations are not reliably reversible.
EvoUndo: Recoverability-Constrained Self-Evolution for LLM Agent Harnesses verifies recoverability of
model-generated self-modifications — prompts, tools, middleware, resources,
execution harness — across counterfactual states, on the observation that a
mutation made in one state may not be undoable from another. Across 600
one-shot self-evolution tasks, 197 capability-improving mutations fail
recoverability verification, and conventional repair strategies recover 0 of
197 under the original representation. Deterministic oracle analysis recovers
48/197 under the original recovery language and 191/197 under an extended
recovery calculus; a protocol-locked 2×2 attributes the difference to exact
state-address grounding (0/48 → 38/48, 79.2%) and recovery-language
expressivity (142/143, 99.3% in the oracle-defined S1 stratum). On
gpt-oss-120b, combining both reduces recovery to 133/143 (93.0%); a
Qwen3.8-27B replication does not reproduce that interaction
(source).
The 0/197 is the number this page should carry. Every disclosure convention argued for above — name the scaffold, name the turn cap, name the judge — assumes the harness is a stated configuration. A harness the agent rewrites at runtime is not a configuration you can state, and if the rewrite cannot be undone from an arbitrary state then it cannot be restored for a re-run either. That is a reproducibility failure before it is a safety one: two runs of "the same harness" are not the same harness.
It also gives the comparability problem above a second axis. The LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering framing separates Controller from Worker; EvoUndo points out that the Controller can modify the Worker's harness mid-run, which neither of those roles accounts for.
What is not established: 197/600 is a one-shot rate, and nothing read gives the figure for horizons where mutations compose. The 191/197 is an oracle result with no runtime, token or engineering cost attached. And "model-dependent" is a description of the negative interaction, not an explanation of it.
The day's other harness datapoint, recorded for what it does not contain
Claude Fable 5.1 shipped on 2026-09-01 with Terminal-Bench-Science 0.1 at 52.6% against Claude Fable 5's 24.7%, Claude Opus 5's 29.0% and GPT-5.6 Sol (and Terra, Luna)'s 22.4% (source). It is a vendor-run agentic benchmark at version 0.1 with no independent replication, no harness disclosure, and no published turn cap, scaffold or judge — the exact shape this page has spent August arguing is a claim about a configuration rather than a capability ordering. Recorded as such, not as a ranking.
2026-09-01 — the harness stops being the nuisance variable and becomes the subject
Every entry above treats the harness as something to disclose: name the scaffold, name the turn cap, name the judge, so that two numbers can be compared. LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering inverts the frame. It holds the coding agent fixed and evaluates the model that drives it, on the grounds that "the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task" (source).
The vocabulary is worth recording because the practice now has a name: Loop Engineering — organising development work around a coding agent by designing a loop that monitors progress, assigns work, runs checks and decides what the agent does next, instead of writing each prompt. The named failure modes are the ones this page has been describing as configuration noise: a loop that trusts a stale progress note, skips needed verification, spends its budget in the wrong direction, or stops before the task is safe to submit.
| Role | Status in LoopArena |
|---|---|
| Controller | the model under evaluation |
| Worker | a separate, fixed coding agent |
| Three settings trade execution scope against cost: Type I scores next-step | |
| Loop Contract selection through execution-validated questions **without | |
| running the Worker**; Type II controls a slice of a full task; Type III | |
| runs the paired full task from its original state. |
| Figure | Value |
|---|---|
| Best observed Strict Success Rate, full tasks | 24.69% |
| Mean paired reduction in estimated inference cost | 64.4% |
| Type II vs Core criterion, rank agreement | Spearman's ρ = 0.9747 |
| The ρ figure is a rank claim and licenses nothing about scores. Type II | |
| reproduces Type III's ordering cheaply; quoting a Type II number as a Type III | |
| number would be exactly the substitution this page exists to catch. |
Why this is the page's own argument arriving from the other side. The 2026-08-21 entry recorded a 6.8-point spread attributable to the harness before training starts, and the 2026-08-27 entry recorded the harness becoming a trained artefact. If the harness moves the score by that much, then the harness's controller is a measurable capability rather than a confound — and LoopArena is the first benchmark here to score it directly. It is the second paper to measure harness sensitivity instead of being confounded by it, after Training Agents to Evolve with Their Harness: TaoLive Digital Avatar Agent Technical Report (arXiv:2608.15763); the two approach the variable from opposite ends and neither cites the other.
The gap that keeps this from being a comparability fix. LoopArena names no model anywhere in its abstract — not the best Controller, not the fixed Worker. A benchmark built to disentangle two systems, published without identifying either, cannot be tracked across releases, which is the one thing a benchmark exists to enable. The 24.69% is recorded here attributed to nobody.
2026-09-03 — the judge has a ceiling, and a vendor's own release splits along the harness seam
Two things landed today that measure the same seam from opposite ends: a benchmark that puts a hard number on how well the harness's scorer works, and a vendor release whose two coding benchmarks moved 9.2 points apart.
AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling measures the judge. 3,808 instances over six workflow-DAG topologies and three difficulty tiers, five generators, six judges from 20B to frontier scale, run paired with and without ground truth. On hard queries without ground truth, all six judges converge into a 77–82% alignment band regardless of scale (source). Alignment degrades monotonically with difficulty, 1.5× faster without ground truth.
That is the first ceiling this page can quote for a harness component. Every result on this page about spread between harnesses has assumed the scorer is a fixed point; on hard agentic tool-calling it is not, and it does not become one by spending more parameters.
Three sub-findings change practice rather than framing:
- Ground truth can hurt. Exposure reduces alignment for GPT-5.4 by 1.5 pp and Gemini-2.5-Pro by 3.9 pp, read as over-anchoring. The assumption that handing a judge the reference can only help is wrong for two of the models tested.
- The obvious knobs do nothing. Chain-of-thought reasoning and judge temperature both have negligible effect. Structured rubrics give up to +6.5 pp but "do not generalize uniformly across judge–generator pairs" — which is a per-pair tuning problem, not a rule.
- "Best judge" has no scorer-independent answer. With ground truth QwQ-32B best matches the programmatic reference; a human validation study names GPT-OSS-120B as most human-aligned. A 32B model and a 120B model win under two yardsticks, and most published agentic evaluation reports one judge and one number.
Gemini 3.8 Flash shows the seam from the vendor's side. In one release, against its own predecessor: Terminal-Bench 2.1 81.6% → 90.8% (+9.2) and SWE-bench Pro 60.4% → 61.6% (+1.2) (source). Both are coding benchmarks. Terminal-Bench scores an agent completing a task end to end — invoking tools, recovering from its own errors — and SWE-bench Pro scores a patch. A release that moves one nearly eight times as far as the other is evidence about the operating loop, not about coding knowledge, and it is the cleanest instance of this page's thesis yet published inside a single vendor's own comparison table.
Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement is the same claim as a method. A framework layered above three existing harnesses — Codex/GPT-5.5, OpenCode/DeepSeek-V4-Pro, Pi/MiniMax-M3 — improves all three, average relative gain 52.25%, maximum 82.86% after three iterations (source). Models fixed, only the layer above the harness varies.
And here the two findings collide, which is worth stating rather than smoothing. HoH's gains are scored on GameCraft-Bench, FrontierSWE and ProgramBench. Nothing read says how those benchmarks score their outputs. If any of them uses an LLM judge on hard agentic tasks, AgentJudgeBench's 77–82% band is inside the same measurement — and whether 52.25% sits above that band or within it cannot be determined from either paper. Neither cites the other. The pairing is this wiki's, and it is an open question, not a refutation.
One more version-hygiene note, from today's Alibaba capture. Coverage of Qwen 3.8 Max's 0902 snapshot carries Terminal-Bench 2.1 = 86.6 and TerminalBench 3.0 = 11.3 → 29.0 in the same articles (source). Those are not a regression and not a contradiction: they are two different benchmarks with similar names. Google's 3.8 Flash figure is Terminal-Bench 2.1. Any page here quoting a Terminal-Bench number must carry its version string, or the next comparison drawn across two of them will be arithmetic on unrelated scales.
2026-09-04 — three papers make the harness the object, and a frontier launch attaches a harness condition to its headline number
Four datapoints in one day, and they do not point the same way.
Three papers in one snapshot ask whether the model can build its own harness, and none of them cites the others (source):
| Paper | What it does to the harness | Result |
|---|---|---|
| HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? | scores the harness the model builds, weights fixed | generated harnesses substantially behind human-engineered references on code and search/research; match or exceed on writing and ML experimentation; Evolution gains unstable, transfer partial, and dependent on the model executing the harness |
| Aspire: Can Models Self-Evolve from Vague Goals? | hides the metric — a vague goal, held-out 520 items, weight and harness evolution | loops complete; weight gains sparse and unstable; best evolved harness below the Qwen-Agent reference; continued search can erase earlier improvements |
| Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills | leaves the harness alone, moves operational knowledge into a separate layer | +134.3% MLE-bench, +34.4% PaperBench, +9.2% FrontierCS, +14.0% PassNet with backbone, harness and budget held fixed |
| **Read together, the three say the same thing from opposite sides: the harness is | ||
| the hard part, and the model is not yet good at writing it.** HarnessDev and | ||
| Aspire both find generated infrastructure losing to a human-engineered reference | ||
| and both find the improvement loop unstable — one with a concrete objective, one | ||
| with a deliberately vague one, which rules out "the goal was underspecified" as | ||
| the explanation. Repo-To-Skill takes the other route: it holds the harness fixed | ||
| and supplies what the harness was missing — 5,000+ verified skills distilled | ||
| from 1,000 repositories — and gets the largest reported gains of the three. |
This qualifies Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement from the day before, which reported +52.25% average relative by wrapping existing harnesses in an improvement loop. HoH improves a mature harness; HarnessDev builds one from a seed and finds it behind. Both can be true, and together they locate the gain in iteration over engineered scaffolding rather than in the model's ability to produce scaffolding. Neither paper says this; the pairing is this wiki's.
And EvoUndo: Recoverability-Constrained Self-Evolution for LLM Agent Harnesses (09-02) supplies the cost. It found conventional repair recovering 0/197 capability-improving self-modifications. Aspire's "continued search can erase earlier improvements" is that same irreversibility observed as a performance curve rather than as a recovery rate.
The launch datapoint, and it is the sharpest one this page has recorded
GPT-6 Astra shipped with ARC-AGI-3 carried at two values, 98.6% and 99.9% — and the higher figure comes with a condition stated in the coverage: it holds under OpenAI's own provider-adapter harness, and stateless API calls are said to score far lower (source).
The comparison offered alongside it is 7.8% for GPT-5.6 Sol (and Terra, Luna) — and 7.8% is already on this page, as the ARC Prize-verified figure that OpenAI's own re-run put at 13.3%. So the headline margin is between a number produced under a vendor's stateful harness and a number produced under a third party's, and the two are not the same measurement. A ~92-point gap is not evidence about a model until both sides are known to have been run the same way, and nothing read says they were.
This is the cleanest example the page holds of its own thesis appearing inside a single vendor announcement: the vendor disclosed the harness dependency, which is more than most do, and the disclosure is what makes the number uninterpretable as a cross-model comparison.
Two more figures from the same week that cannot be compared, and are not
- DeepSWE v1.1: Astra 74.1% (2026-09-03) against Muse Spark 1.3's 75.4 (2026-09-02). Two vendors, two announcements, no shared harness read anywhere
- Muse Spark 1.3's own scorecard compares 1.3
maxagainst 1.2xhigh— different reasoning tiers — andmaxis not the mode that shipped. The 55.0 → 75.4 jump is therefore partly a tier change, and nothing read labels which rows are which tier. A reasoning tier is a harness parameter by another name
2026-09-05 — the prompt is part of the harness, and it reorders models
Two papers in one snapshot, and between them they move this page's variable in both directions — one adds a degree of freedom nobody reports, the other says the harness cannot be optimised apart from the weights.
RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests measures the gap between how benchmark problems are written and how users actually write, and finds it is not a stylistic detail. Against real prompts from SWE-chat: requests carrying only a problem statement are 88% of real prompts and 7% of benchmark problems, and 87% of real prompts are casual against 94% of benchmark problems being formal. The benchmark is 381 multi-variant task families derived from SWE-bench Verified and Pro, where variants share the same task and the same gold patch and differ only in information composition and linguistic style — which makes the prompt the single moving part. Seven models, and realistic inputs drop resolution rates 6.4 pp on average and can change model rankings (source).
The ranking result is the one that belongs on this page and the register result is the one worth not misreading. Style has only small, model-dependent effects; what matters is which information categories the prompt contains, and the two that matter — Desired Behavior and Motivation — are the two real prompts most often omit, while Environment Information and Reproduction Steps "merely add tokens without measurable benefit". So the finding is not "users write badly". It is that the prompt distribution is a configuration variable, it is not reported by anyone, and it does not preserve order.
6.4 pp is small next to what this page already holds — ARC-AGI-3 spanning 7.8% → 38.3% on GPT-5.6 Sol (and Terra, Luna) from harness settings alone, and Astra's 98.6% / 99.9% split conditioned on a vendor's own provider-adapter harness. It is also the one that applies to every SWE-bench figure on this wiki, including Claude Opus 5's 96% Verified and Claude Fable 5's 80.3% Pro. It does not invalidate them: all were measured on the formal distribution, consistently. It says the distribution is not the deployment one and the ordering is not guaranteed to survive the change.
WHALE: A Simple Recipe for Joint Harness-Weight Optimization attacks the same seam from training. Weights and harness are optimised by alternation — update weights under the current harness, then search for a harness under the updated weights — beating weight-only, harness-only and Fast-Slow Training by 4.15–24.38 pp on Qwen3.5-2B/4B across search QA, math and chess puzzles. The row that matters here: harness search matches peak weight-only accuracy with far fewer rollouts on SearchQA, and improves math accuracy only after a weight update. Either component can be the bottleneck, and which one it is changes by domain.
If the pair must be optimised jointly, it must be reported jointly. That is this page's thesis stated from the training side rather than the measurement side, and it is the first result read here that supports it with a controlled three-way comparison rather than by observing spread. It also sharpens the comparability problem below: a cross-lab weights comparison at fixed harness is not measuring a fixed object, because the harness each lab converged on is co-adapted to its own weights.
Neither paper cites the other, and neither cites the harness cluster of 2026-09-03/04. The grouping is this wiki's.
2026-09-08 — the argument is adopted by the people it would embarrass, and a controlled study measures the harness gap directly
Two papers, and the first is the one this page has been waiting for.
Iris (2026-09-03) trains two open-source search agents at 35B-A3B and 397B-A17B and reports BrowseComp 82.2 / 88.6, BrowseComp-ZH 84.8 / 85.1, DeepSearchQA 86.9 / 92.9, HLE 52.3 / 56.4. What matters here is the sentence attached to those numbers: the authors judge that inference-time context management is worth more on these benchmarks than most reported differences between systems, and therefore evaluate every benchmark both with and without it, holding the tool set, the context limit and the judge fixed (source).
This page has been assembling its case largely from outside the results being compared — from a 7.8%→38.3% spread on ARC-AGI-3, from a 0.00% published rate against a 60–80% demonstrated one, from ablations that disagree. This is the first frontier-scale result in this wiki whose own authors reach the same conclusion about their own numbers and change their reporting because of it. The scope declaration that follows — a single ReAct agent, no sub-agents, no test-time verification — does the same work from the other end: it says how much of the score cannot be orchestration.
The catch is that the figures published are the enabled condition only. The snapshot carries no without-management numbers, so the size of the effect the paper exists to highlight is unread here. A paper can adopt the argument and still publish only one half of the comparison.
Select, Compress, Reinvest (2026-09-03) supplies what Iris withholds: a measurement of the gap itself. It holds the frame scorer, the prompt boundary, the resolution policy and the answering model fixed and varies one decision at a time across six training-free selection rules, three long-video benchmarks and two answering models. Selection is the largest lever — eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points on LongVideoBench's hour-long bin — and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector across all three benchmarks (source).
Then the number this page will cite it for: the authors report a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget, alongside an implementation bug they found in their own baseline. That is this page's thesis stated as a measured interval — the same rules, the same budget, two harnesses, and up to 3.74 points of difference attributable to neither the model nor the method. It is why, in the authors' own words, these comparisons have to happen inside one controlled harness rather than across papers.
A third paper this run reaches the same seam from underneath. AutoTraceGT finds that hand-built failure taxonomies are 9–27% incomplete. Every paper in this cluster varies the environment, the harness, the prompt distribution or the scoring function and still reports a score; that one asks whether the categories the score is bucketed into were right to begin with.
2026-09-16 — the specification channel itself becomes the variable, and the leaders separate by 20 points once it is turned up
Every entry above varies the environment, the scaffold, the prompt or the scorer. ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks varies how the task is specified: instead of an issue text, the agent is given a working reference application and must infer the behaviour by interacting with it, then implement it in an incomplete one. 1,975 replay-verified behaviors across 26 applications, 4,063 tasks, constructed with no human intervention (source).
GPT-6 Astra 49.2%, Claude Opus 5 28.8% on cumulative workflows in full-application reconstruction. No harness is named on either side, which puts a 20.4-point frontier-model gap on this page under exactly the disclosure condition the rest of it exists to record.
The difficulty dial is the contribution for this page. "Restoration depth" controls how much of the target application has been removed, and the published curves run 100% → 64.0% and 96% → 32% from depth 1 to depth 8. At depth 1 both agents are at or near ceiling and the benchmark separates nothing; at depth 8 it separates them by 32 points. That is a harness parameter deciding whether a benchmark discriminates at all — the same property this page recorded for context budget on 09-04 and for prompt phrasing on 09-05, now for how much of the problem is left standing.
What is not established: the abstract does not say which of the nine agents each depth curve belongs to, so neither curve is attributed here; there is no aggregate score across the 4,063 tasks; and nothing states whether the harness was held constant across the nine agents, without which the 49.2 / 28.8 comparison is not checkable.
2026-09-23/24 — the first cross-vendor score of the September wave is published by one of the vendors, and a benchmark starts measuring the fork instead of the run
Three things landed inside 48 hours and they pull in opposite directions.
One. The 2026-09-23 ingest recorded that the three frontier releases of 2026-09-22 — Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna — shared no benchmark at all, leaving only structural comparisons. The first figure spanning them arrived the next day, in OpenAI's MentalHealthBench: GPT-6 Astra 57.3%, GPT-6 Sol 53.9%, Claude Opus 5.5 52.4%, GPT-6 Luna 50.2% (source).
The publisher holds three of the top four rows. That is not a reason to discard the numbers, and this page has never treated a vendor benchmark that way. It is a reason to note which part of the release does the work: OpenAI published the methodology and the synthetic data, so the figures are reproducible by someone else, which is more than almost any vendor table this page records. No effort level, run count, variance or confidence interval is published for any row — so the 1.5-point gap between Sol and Opus 5.5 is a gap this page cannot read as an ordering.
The comparison set is also not contemporaneous: GPT-4o (March 2025) 32.1% and Gemini 2.5 Pro 29.5% sit beside four models under six weeks old, and no current Gemini and no open-weights model appears at all.
Two. Grok 4.7 shipped on 2026-09-21 with CursorBench 4.0 46.3%, EEBench 64.0%, Harvey Legal Agent Benchmark 19.6% and no harness, effort level, run count or variance named for any row (source). Its comparison set is GPT-5.6 Sol and Fable 5.1 — superseded the following day. So the September wave now contains a model measured only against the previous generation, and a benchmark measuring the current generation published by a competitor. Neither is comparable to the other, which is this page's subject stated in its plainest form yet.
The Harvey figure is the one to be careful with: 19.6% against GPT-5.6 Sol's 2.5% is a 7.8× ratio on a benchmark where the whole field scores under 20%, and nothing read names the harness that produced it.
Three, and it is the constructive one. The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks stops measuring runs and starts measuring decision forks inside them — mined automatically from parallel attempts at the same task and from detours inside a single trajectory, with no human annotation. The best frontier model answers 59.7%, and a larger reasoning budget does not improve accuracy.
That construction belongs on this page for a specific reason: a benchmark built out of another benchmark's discarded traces inherits that benchmark's harness, and Taste-Bench names none. The measurement is new; the unnamed dependency underneath it is the one this page has been recording since August.
2026-09-18 — the same substitution on five operating systems, and the leader passes everything on 2.8% of tasks
RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents does to computer use what ProgramDistill did to web SWE tasks two days earlier: the specification is a running reference, the agent must discover its behaviour and rebuild it with no prescribed workflow, and the reference then serves as the oracle for hidden behavioural tests. Five platforms — Ubuntu, macOS, Windows, Android, Web — under a unified harness with native GUI control and coding tools, with RecreationBench's 250 held-out tasks carrying reference-grounded programmatic and visual assertions validated on the reference and by human reviewers before the suite was frozen (source).
The result this page wants is the pair of thresholds, published together by the authors: GPT-6 Astra leads at 58.1% overall and passes all programmatic tests on 2.8% of tasks. Same system, same run, a factor of twenty between them. Every agentic figure this page has argued about was a single number with an unstated denominator; here both denominators are stated, and the gap between them is larger than any harness effect recorded above.
The harness is described and not specified. "A unified harness with native GUI control and coding tools" is exactly the disclosure level this page was created to flag, and no comparison model is named, so unlike ProgramDistill there is no margin between two frontier agents to record — only a leader.
Three qualitative findings, none with a figure: agents reproduce static interface structure more reliably than interactions and computed outputs; generated applications remain smaller and more monolithic than their references; and models trained on these trajectories improve on five out-of-distribution benchmarks that are not named, while more frequently verifying their rendered outputs.
The construction side of the same move landed the same day. CodeMidas: Scaling Agentic Coding RL Environments from Code Itself synthesises RL training environments from source code alone — 5,545 tasks from 3,185 codebases across 23 languages — which makes the artefact-as-specification substitution a training technique as well as an evaluation one. Recorded on Agentic Reinforcement Learning with its figures; noted here because it means a harness this page would ask to have disclosed may now be part of the training data rather than only part of the measurement.
2026-09-25 — the repository is a variable too, and the agents were partly reading cues rather than code
Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It? makes the test repository an evaluation-time latent variable and reports that removing familiarity — not difficulty — consistently degrades agent performance and substantially increases interaction cost on SWE-bench Verified and SWE-QA (source). SchrodingerRepo instantiates a behaviourally identical repository at the moment the agent enters the environment, eroding cues through problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting. The additional cost falls on exploration and localization.
This is the page's argument extended one level outward. Everything recorded here since August treats the harness as the unreported variable behind a score. This paper says the codebase the harness runs on is one too, and it is a variable that models may have memorized. The conclusion it draws — current coding agents "may partially rely on memorized repository-side cues" — is the first testable version of the contamination objection this page has recorded.
It lands on a specific, live gap. On 2026-09-22 Claude Opus 5.5 shipped with no SWE-bench-family row at all and GPT-6 Sol and GPT-6 Luna shipped with DeepSWE v1.1 as their only benchmark, which the 2026-09-23 ingest recorded as vendors sharing no scale. This supplies a reason a vendor might move off SWE-bench that is not "we score worse on it" — and it also undermines the replacement, because DeepSWE is built from the same kind of repository. Nothing read applies SchrodingerRepo to DeepSWE.
The cost half is the part this wiki should carry forward. Every per-task cost ratio on this wiki — Sol at "~80% below Fable 5", Luna at "93% below Claude Opus 5" — was measured on canonical repositories. If unfamiliarity raises interaction cost, those ratios are a property of the measurement repository as much as of the model. Neither vendor states which repositories, and this paper gives no multiplier, so the observation cannot yet be quantified in either direction.
Its own record is entirely directional: no percentage-point drop, no cost multiplier, no model named, no ablation across the four transformation levels, and no harness named — the last being this page's standing finding, reproduced by a paper about measurement hygiene.
Agensh: Scaling Organizational Intelligence to 1,024 Agents arrives in the same snapshot pointing the other way, and the pair is why both are here. Agensh removes the central orchestrator and scales a harness to 1,024 agents: on the five hardest ProgramBench tasks with GPT-5.6-sol (high), 1 → 128 agents takes the mean final test-pass rate from 19.31% to 28.78%, and on pandoc alone, 1 → 1,024 agents takes it from 33.89% to 55.06% (source). Set against Harness-Zero: Harness Distillation via Agent-as-Harness from two days earlier — 23.3% → 44.3% with the harness removed, against 41.7% with one attached — the field is now publishing, inside 48 hours, that the scaffold is the constraint and that the scaffold should be an organization and made a thousand times bigger. The two are measured on different benchmarks with different models and neither cites the other, which is the comparability problem stated at the top of this page, arriving now between two papers rather than between two vendors.
Agensh reports no cost axis at all — no tokens, no dollars, no wall-clock — against a paper whose stated justification is latency, and its 1,024-agent headline rests on a single task.
2026-09-28/29 — a version number moves one model 70 points, and a vendor compares two of its own models at different defaults
Claude Sonnet 5.5's launch table gives Claude Sonnet 5 10.3% on Terminal-Bench 4.0. This wiki holds the same model at 80.4% on Terminal-Bench 2.1 (source).
Both figures are correct, and that is the finding. A 70.1-point spread on one unchanged model is produced entirely by the harness two major versions apart. Nothing about the model moved between those two numbers; the thing being measured did.
This is the cleanest instance this page holds, and it is cleaner than the cross-vendor cases above for one reason: there is no vendor to suspect. The same lab published both numbers about the same model, so no selection, no favourable setting and no benchmark choice explains it. A benchmark name with a version suffix is not a series that trends — it is a set of distinct instruments that share a brand, and a reader who watches "Terminal-Bench" across this wiki's model pages is watching four unrelated scales.
The second, smaller problem in the same table
The table's two Anthropic columns are not stated to be the same effort setting, and the
two models do not default to the same one: claude-sonnet-5-5 defaults to high,
claude-opus-5-5 to medium
(docs). No row in the launch table
names an effort level for any model.
So the headline result — Sonnet 5.5 at 70.6% over Opus 5.5 at 66.4% on Terminal-Bench 4.0 — is a comparison whose most likely confound is published in the vendor's own docs and omitted from the vendor's own table. The 2026-09-22 Opus 5.5 entry above already recorded that no effort level was stated for any of its nine rows; six days later the same omission now sits across a pair of models with different defaults, which converts a reporting gap into a possible ranking error.
What this does not say: it does not say Sonnet 5.5's lead is an artefact. It says the table does not contain what would be needed to tell, and that the missing field is one the vendor documents elsewhere. Anthropic's own guidance still recommends Opus 5.5 over Sonnet 5.5 as the default, which points the opposite way from the row.
Why it lands here rather than on the model page
The relevant unit is the instrument, not the release. Yesterday's 👀 item carried Noam Brown criticising CAIS for evaluating every model at "reasoning high" when "high" is not comparable across models. This is the same objection from the other side: here the two models are not at the same nominal setting, and the setting is not printed. Either way the fix is the one this page has asked for since 2026-07-29 — publish the harness with the number.
2026-10-01 — the same benchmark name carrying two measurements 5x apart, and this is the cleanest case yet
This page's argument has been that a score without a harness is not a measurement. ExploitBench is now the demonstration.
Two parties published ExploitBench figures for overlapping model sets:
| Model | Z.ai model card | Anthropic Frontier Red Team |
|---|---|---|
| GLM-5.3 | 54.4 | 12% (50 / 410) |
| GLM-5.2 | 24.4 | at or near 0% |
| Claude Opus 4.8 / 4.6 | 40.0 (4.8) | at or near 0% (4.6) |
| Kimi K3 | 32.2 | at or near 0% |
| Fable 5 | 78.0 | — |
| Claude Mythos Preview | — | 14% (56 / 410) |
| Sources: (Z.ai card), | ||
| (Anthropic). |
What is established. Z.ai's footnote names Claude Code 2.1.207 as the harness for its ExploitBench row, with sampling parameters, context length, timeout, turn cap and container policy stated. Anthropic reports a denominator — 410 attempts — and names no harness at all. So the better-documented figure is the vendor's own, on its own model, which is the opposite of the usual arrangement.
What is not established. Whether the task sets are the same benchmark; whether
54.4 is a percentage (this wiki has written it both ways); and which is the better
estimate. The rank order survives both runs — an Anthropic frontier model
narrowly above GLM-5.3 — while GLM-5.2 moves from 24.4 to ~0 and Opus from 40.0 to
~0. A difference that preserves ordering while collapsing magnitude is the
signature of a different pass criterion, not a different model.
Why this is worse than the cases already on this page. Every earlier entry here
is a comparability problem: two harnesses, two scores, and a reader who cannot
line them up. This is a countability problem. 54.4 and 12% are both read as
"GLM-5.3 on ExploitBench" by anything that parses a benchmark name — including
scripts/claim-check.py, which compares a row label and a column header against a
cited snapshot and would find both faithful to their own sources. Two true rows,
one name, and no check in this repository can see the collision.
Recorded on GLM-5.3's ## Conflicting Reports per the schema, unresolved
by design, and carried to the W40 lint as a check the tooling does not perform.
2026-10-05 — two independent entries land within 2 points of a frontier model on ARC-AGI-3, and nothing establishes that the scores are the same measurement
This is the comparability problem arriving from a direction this page has not had yet: not two harnesses for one model, but two populations of solver sharing one benchmark name.
This wiki already holds three ARC-AGI-3 figures, all described on their pages as ARC Prize's own standardized harness: 30.2% for Claude Opus 5, 7.8% for GPT-5.6 Sol (and Terra, Luna) — against 13.3% as re-run by OpenAI, recorded on this page as unreconciled — and the disputed 98.6% / 99.9% pair on Astra.
The ARC Prize 2026 Kaggle competition track reports, for its Milestone #2 with a 2026-09-30 cut-off, announced 2026-10-01:
| Entrant | Final score | Prize |
|---|---|---|
| Daniel Franzen (1st) | 27.9% | $25,000 |
| Lord Han Solo (2nd) | 23.8% | $7,500 |
| Lohit Siriki (3rd) | 22.5% | $5,000 |
| Award was conditional on the solutions being open-sourced. A later ARC Prize post | ||
| on X reports a 28.34% high score by Yi-Chia Chen taking first place; the post's | ||
| date is not in anything read. Total ARC-AGI-3 pool $850,000, with a $700,000 | ||
| bonus pool at 100% human-level | ||
| (source). |
What is established. Two open-sourced independent entries score 27.9% and 28.34% on a benchmark where this wiki's highest ARC-Prize-verified frontier figure is 30.2%.
What is not established, and it is the whole question. Nothing read says a Kaggle submission is scored on the same task split, under the same compute budget, by the same harness as the model figures. The Kaggle track awards bespoke open-sourced programs; the 30.2% is a general-purpose model under ARC Prize's model harness. A 2.3-point gap between those two is not a 2.3-point gap, and this page's entire argument is that writing it as one is the error. The figures are recorded side by side and are not combined on any model page.
Provenance, stated because it bounds everything above. arcprize.org,
www.kaggle.com, llm-stats.com and x.com all answer EGRESS_BLOCKED from this
run, so no page was fetched: the figures come from two independent WebSearch
passes that agree on every value. This is the Import AI 474 route of 2026-10-01 and
carries the same limit — reported, not read.
One claim is refused. The trail started at prefetch candidate #17, an r/MachineLearning post of 2026-10-04 titled "Top ARC-AGI-3 scores on Kaggle just went from 7% to 56%". Both passes looked for the 56%; neither found it, and the highest figure either returned is 28.34%. Not adopted, and the mismatch is recorded rather than resolved.
A structural gap this surfaces. ARC Prize is the verifying authority behind
figures on at least four pages here — Claude Opus 5,
GPT-5.6 Sol (and Terra, Luna), Astra and Context Compaction — and it
has no entity page, no sources.yaml entry at any tier, and no snapshot in
sources/evals/. The 2026-10-01 milestone was published four days before this run
read about it secondhand. This is the Runway / Ant Group (inclusionAI / AntLing)
shape of 2026-09-26: a body this wiki cites repeatedly and polls never.
The same day's benchmark paper says the answer keys are wrong
Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows (arXiv 2610.02122, HuggingFace Daily Papers 2026-10-05, 26 upvotes) builds a 210-task agentic analytics benchmark on a 235-table, 7.5-billion-row simulated ERP warehouse, and justifies doing so with a claim about the incumbents: established text-to-SQL benchmarks "evaluate query generation alone, and audits have found their answer keys frequently wrong" (source).
Which audits and which benchmarks is not stated in anything read, so nothing here is corrected on the strength of it — the same treatment this page gave SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents, whose finding was specific and whose per-model corrections were not published either.
What Argo-Bench grades is the part that belongs here: the agent files actions — banning accounts, allocating budgets, issuing back pay — and "the grader scores each by its consequences in the simulator", with the simulator's ground-truth state withheld from the warehouse the agent sees. The strongest of 14 frontier and open-weight models scores ≥95 on only 34.8% of tasks and averages 59.5 points. No model is named, so no figure attaches to any model page here, and the harness is unspecified — which by this page's own standard means 34.8% is not yet comparable to anything.
And a decoding rule moved a reasoning score 11 points with the weights frozen
Decoding Looped Transformers Better for (Almost) Free reports AIME 2024 pass@1 61.88% → 73.33% on Ouro-2.6B-Thinking from a training-free change to how a looped model's intermediate passes are read, and that halving the loop count then matches full-depth baselines (source). Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It reports 15.5% → 99% on 24-link reference chains from a rank-8 LoRA at one layer with every other weight frozen (source).
Both are the same degree of freedom as the ones already on this page, moved inward.
A looped model's published score is underdetermined without its decoding rule and
loop count; a base model's published score on a compositional task is a statement
about the default forward path and not about the weights, and on this task the two
differ by 83.5 points. Neither paper was read — arxiv.org is blocked — and
neither names a model this wiki holds a page for.
Practical consequence for this wiki
Benchmark rows on model pages record the figure with the harness or source that produced it
where that is known, and a disputed figure goes to ## Conflicting Reports rather than being
silently reconciled. The two "official harness" numbers for GPT-5.6 Sol — 7.8% verified by ARC
Prize, 13.3% as re-run by OpenAI — are recorded as disagreeing because nothing read reconciles
them.
The same discipline applies to the leaderboard snapshots this repo scrapes: LMArena is quoted as a percentage with its confidence interval rather than as Elo, and Artificial Analysis figures are quoted with the column heading above them (source).
Key Sources
- 2026-09-08 — the benchmark this wiki quotes on 24 model pages has been audited and found to leak. SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents reports SWE-Bench Pro's evaluation undermined by reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and by task quality issues — misleading problem statements and improperly scoped tests. The released SWE-Bench Pro Verified adds anti-hacking safeguards and minimally corrects flawed instances, and on it "some models perform substantially worse than previously reported". Which models, and by how much, is not published in anything read, so nothing on this wiki is corrected on the strength of it — what changes is what a SWE-bench Pro cell means (source)
- OpenAI, "How enabling two settings tripled our scores on the ARC-AGI-3 benchmark" (2026-07-29) (source)
- ARC Prize results — Claude Opus 5
- ARC Prize, analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3
- LMArena snapshot (source)
- Artificial Analysis snapshot (source)
- Qwen3.8-27B release — SWE-MM on the Claude Code harness (2026-08-14) (source)
- GLM-5.3 release — four benchmarks, no harness (2026-08-14) (source)
- AI4AI at Test-Time, arXiv:2608.12307 (source)
- HuggingFace Daily Papers, 2026-08-22 — EnvHarness, FACET, SWE-bench Science, FM-Bench and MemTrapBench (source)
- HuggingFace Daily Papers, 2026-08-20 — Agent Skills, Harness the Memory, Agent Lightning v1.0, Agentic ESOpt and ASI-Bench (source)
- HuggingFace Daily Papers, 2026-08-19 — StateM, AutoResearchEval, ClawGym II and Ventor-QTest (source)
- HuggingFace Daily Papers, 2026-08-26 — Prime Agent, Thinkingbox, Apodex 1.1 and AgentMercury (source)
- HuggingFace Daily Papers, 2026-08-16 — DarwinX, AutoDesign, SHAPER and SkillZip (source)
- HuggingFace Daily Papers, 2026-08-27 — JIT-Agent, AutoSaddler, Recuris, Meta^n and FrontierChallenge (source)
- HuggingFace Daily Papers, 2026-08-28 — SWE Refactor Bench, PILOT and TaoLive HAT (source)
- GLM-5.3 open-weights model card — 16 benchmark rows with a named harness per row (2026-08-28) (source)
- Embrace The Red, "Breaking Claude Code Opus 5 Auto Mode" (2026-08-26) — a 0.00% published rate and a 60–80% demonstrated one (source)
Related Concepts
- Reasoning Models
- Test-Time Compute (Inference-Time Compute Scaling)
- Agents (LLM Agents)
- Conceptual Reasoning Index (CRI)
- Model Routing
- GPT-5.6 Sol (and Terra, Luna)
- Claude Opus 5
- DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545)
- StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089)
- How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905)
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? (arXiv:2608.19799)
- FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents (arXiv:2608.18423)
- ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798)
- Prime Agent: A Self-Improving RLM Harness (arXiv:2608.23552)
- One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows (arXiv:2608.19741)
- Apodex 1.1: Scaling Agentic Intelligence for Complex Work (arXiv:2608.23283)
- Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (arXiv:2608.16391)
- Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See (arXiv:2608.17744)
- Decision-Metric Alignment in Latent World Models: Diagnostics and Action-Conditioned Objectives for MPC Planning (arXiv:2608.18746)
- SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation (arXiv:2608.18701)
- SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation (arXiv:2608.17426)
- JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution (arXiv:2608.25593)
- AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces (arXiv:2608.23041)
- Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses (arXiv:2608.24876)
- Meta^n: Recursive Self-Improvement through Emergent Depth (arXiv:2608.24735)
- FrontierChallenge: Evaluating Scientific Workflow Completion (arXiv:2608.24979)
- AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
- Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
- HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- Aspire: Can Models Self-Evolve from Vague Goals?
- Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
- Gemini 3.8 Flash
- Astra
- Muse Spark 1.3
- Last Translation Benchmark
- Iris: Climbing to the Search Frontier
- Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs
- Using Grounded Theory for Agent Behavior Analysis at Scale
- Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?
- Agensh: Scaling Organizational Intelligence to 1,024 Agents
Referenced by
Sources
- sources/blogs/arcprize-2026-10-01-arc-agi-3-milestone-2.md
- sources/arxiv/2026-10-05/2610.02122-argo-bench.md
- sources/arxiv/2026-10-05/2610.02185-loopcd-looped-decoding.md
- sources/arxiv/2026-10-05/2609.36585-stop-thinking-too-early-lora.md
- sources/papers-daily/hf-daily-2026-10-05.md
- sources/blogs/anthropic-2026-09-29-glm-5-3-cyber-capabilities.md
- sources/papers-daily/hf-daily-2026-09-25.md
- sources/blogs/openai-2026-09-23-mentalhealthbench.md
- sources/blogs/xai-2026-09-21-grok-4-7.md
- sources/papers-daily/hf-daily-2026-09-24.md
- sources/papers-daily/hf-daily-2026-09-22.md
- sources/papers-daily/hf-daily-2026-09-21.md
- sources/papers-daily/hf-daily-2026-09-11.md
- sources/papers-daily/hf-daily-2026-09-08.md
- sources/papers-daily/hf-daily-2026-09-06.md
- sources/blogs/openai-2026-09-03-gpt-6-astra-system-card.md
- sources/papers-daily/hf-daily-2026-09-05.md
- sources/papers-daily/hf-daily-2026-09-04.md
- sources/blogs/openai-2026-09-03-gpt-6-astra-launch.md
- sources/blogs/meta-2026-09-02-muse-spark-1-3.md
- sources/papers-daily/hf-daily-2026-09-03.md
- sources/blogs/google-deepmind-2026-09-02-gemini-3-8-flash.md
- sources/blogs/alibaba-2026-09-02-qwen-3-8-max-0902.md
- sources/papers-daily/hf-daily-2026-09-02.md
- sources/blogs/anthropic-2026-09-01-claude-fable-5-1-mythos-5-1.md
- sources/papers-daily/hf-daily-2026-09-01.md
- sources/blogs/zai-2026-08-28-glm-5-3-open-weights.md
- sources/papers-daily/hf-daily-2026-08-28.md
- sources/blogs/embracethered-2026-08-26-claude-code-auto-mode-bypass.md
- sources/blogs/deepmind-2026-08-27-double-blind-evaluations.md
- sources/papers-daily/hf-daily-2026-08-27.md
- sources/papers-daily/hf-daily-2026-08-26.md
- sources/papers-daily/hf-daily-2026-08-25.md
- sources/papers-daily/hf-daily-2026-08-23.md
- sources/blogs/huggingface-2026-08-21-asr-benchmark-optimization.md
- sources/evals/artificial-analysis-2026-08-23.md
- sources/papers-daily/hf-daily-2026-08-20.md
- sources/papers-daily/hf-daily-2026-08-19.md
- sources/papers-daily/hf-daily-2026-08-18.md
- sources/papers-daily/hf-daily-2026-08-17.md
- sources/blogs/alibaba-2026-08-14-qwen-3-8-27b-released.md
- sources/blogs/zai-2026-08-14-glm-5-3.md
- sources/arxiv/2026-08-15/2608.12307-ai4ai-test-time.md
- sources/blogs/openai-2026-07-29-arc-agi-3-two-settings.md
- sources/evals/lmarena-2026-07-29.md
- sources/evals/artificial-analysis-2026-07-30.md
- sources/blogs/anthropic-2026-09-28-claude-sonnet-5-5.md