Eval Harness Configuration

Definition

The harness is the scaffolding around a model during a benchmark run: how context is carried between steps, whether reasoning traces survive a turn boundary, how tools are exposed, how many attempts are allowed, how results are parsed. It is not part of the model and not part of the benchmark's task set — and on agentic benchmarks it can move the reported score by more than the gap between two model generations.

A benchmark number is therefore a claim about a (model, harness) pair, never about a model alone.

Why It Matters

On multi-step agentic tasks the harness decides what the model remembers. Discard the chain of thought after each action and the model rebuilds its state every turn; preserve it and the same weights behave like a different system.

The 2026-07-29 ARC-AGI-3 episode is the clean demonstration (source):

Configuration	GPT-5.6 Sol
ARC Prize official harness (ARC Prize verified)	7.8%
Official harness, as re-run and reported by OpenAI	13.3%
Responses API + retained reasoning + compaction	38.3%
Same model, same task set, a 4.9× spread from harness settings alone. For scale, the
verified SOTA that Claude Opus 5 set on 2026-07-27 was 30.2%, and the average
human tester scores 48%.

• Retained reasoning preserves the chain of thought between steps.
• Compaction summarizes older context instead of truncating it.

Neither was built for ARC-AGI-3; both are general Responses API settings available to any API user, and OpenAI reports they also cut output tokens 6×.

The comparability problem

Two things are true at once and the tension between them is the whole subject:

1. A setting available to all users, not built for the benchmark, is a legitimate way to run a model. ARC Prize's reported position is that such settings are fair game if properly reported, and co-founder François Chollet is reported as agreeing (source).
2. Official leaderboard scores use one standardized harness without provider-specific settings, precisely so that numbers from different labs remain like-for-like.

So OpenAI's 38.3% and Anthropic's 30.2% cannot be compared. Nothing published states whether Claude Opus 5 was measured with any equivalent state-preserving configuration, and without that the higher number is not evidence of a better model (source).

Open Problems

• An unpublished harness is not a weaker claim than a published one — it is a different kind of claim. DeepSeek V4-Pro-0813 (2026-08-13) reports Terminal Bench 2.1 87.9, DeepSWE 62.7 and CyberGym 83.3, all against its own preview, and DeepSeek has not released the harness, so "the third-party record is empty" (source). The same day, Gemini 3.7 Flash published DeepSWE 65.3 with no harness statement either (source). Two vendors, one benchmark, three points apart, and nothing in either release makes the numbers comparable.
• Reproducibility now has a measured base rate, and it is not high. A community effort reproduced claims from 2,200+ ICML 2026 papers with coding agents over 19 days: 1,221 participants, 6,816 public logbooks, 35,908 claims labelled by an automated judge, 3,978 confirmed, 266 papers fully and 632 partially reproduced without falsification (source). Coverage summarises this as "more than half" — but 266 + 632 = 898 of 2,200+, which is under half; the "more than half" figure counts at least one claim verified per paper, a weaker measure. Both are recorded because the gap between them is the point. Nothing read states how the automated judge was validated, which makes the headline itself a claim about a harness.
• No convention for reporting harness configuration. A score is published as a scalar; the configuration that produced it usually is not.
• Provider-specific settings have no neutral equivalent. Retained reasoning is a Responses API feature. There is no vendor-independent way to grant a competing model the same state-preservation, so "run everyone with their best settings" and "run everyone the same way" are both defensible and give different rankings.
• The gap grows with task length. Harness effects are small on single-turn benchmarks and large on long-horizon agentic ones — which is the direction evals are moving.
• Self-reported runs are unverified by default. ARC Prize verification is a separate step; most published agentic numbers never get one.

2026-08-14/15 — one vendor disclosed a harness, and a paper measured what it is worth

Two additions from the same 24 hours, and they belong together.

The disclosure. Qwen 3.8 27B's model card states that its SWE-MM figure was run on the Claude Code harness, using the public dev split of SWE-bench Multimodal with modifications described in Appendix 8.3 of the Claude Opus 4.7 system card (source). That is a Chinese lab naming a competitor's harness, by document and appendix number, for one of its own headline rows. It is more harness detail than any other release captured this week published for any figure — and it covers one row of four. The Terminal-Bench 2.1, DeepSWE 1.1 and OSWorld-Verified rows on the same card carry none, and whether the Qwen3.6-27B comparison column was re-run under the same harness is unstated.

The measurement. AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (arXiv:2608.12307) reports that a harness built by a stronger model raises a weaker model's average across four Theory-of-Mind benchmarks from 0.49 to 0.91, with no parameter updates (source). If that holds beyond the task family it was measured on, the harness is not a footnote on a benchmark row — it is potentially the larger share of the number.

Read together: the field is publishing model scores while the paper evidence says the unpublished term may dominate them. A benchmark figure without its harness is not a weak claim; it is an unattributed one — the reader cannot tell which of the two systems was measured.

Version numbers are not comparable either. GLM-5.3 reports Terminal-Bench 3.0 (4.6 → 28.3) while Qwen 3.8 27B and DeepSeek V4-Pro-0813 report Terminal-Bench 2.1 (73.0 and 87.9). Same family name, different suite, three releases in one week, and nothing read maps one onto the other.

2026-08-16 — five papers in two days say the harness is the capability

The 08-14/15 pair above was a disclosure and a measurement. What arrived on 2026-08-16 is a method, four times over, from four unrelated groups. Every one of them holds the model frozen and improves the system around it (source):

Paper	What is evolved	Headline
DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545)	a population of harnesses under selection	~+17 points average; Terminal-Bench 2.1 to 84.7%
AutoDesign (arXiv:2608.13560)	a meta-harness optimising a code agent's harness	PosterBench 54.99 → 67.39 across seven configurations
SHAPER (arXiv:2608.11350)	skills + a context-code harness, embodied, train-free	frozen model serves as both planner and optimiser
SkillZip (arXiv:2608.05604)	compression of the skill library itself	+12.2 points over the strongest baseline at 3.46× compression
Together with AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (arXiv:2608.12307) the day before, that is
five results in two days making the same structural claim from four different
directions — coding agents, document generation, embodied control, and skill
retrieval. DarwinX states the thesis outright: *"a frozen model need not be a
fixed agent: harness selection turns evaluation compute into durable capability."*

Why this changes the size of the problem. Until now this page could say a benchmark figure is a claim about a (model, harness) pair whose second term is undisclosed. These papers say the second term is optimisable, cheaply, without touching weights — DarwinX reports +7.7 points on Terminal-Bench 2.1 on a matched base. Put that next to the vendor-stated Terminal-Bench 2.1 figures this wiki holds from a single week — Qwen 3.8 27B 73.0, Claude Sonnet 5 80.4, DeepSeek V4-Pro-0813 87.9, none with a published harness — and a 7.7-point spread no longer distinguishes two models from two harnesses around the same model.

The unresolved half is cost. None of the four papers read here publishes the compute a loop consumes. "Evaluation compute into durable capability" is an exchange rate quoted with one side blank, which is the same shape of omission this page was created to name.

2026-08-18 — two papers attack the measurement instead of proposing another harness

Everything below this section proposes a harness and reports a gain. Today's batch contains the first two entries that treat the measurement as the defective part (source):

Paper	What it measures	Instrument
Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417)	7 frontier models on 36 long-horizon AI-R&D tasks	rule-based within-run metrics — Solution Framing, Execution, Feedback Control — plus controlled experience-reuse comparisons
Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence (arXiv:2608.11341)	problems that "rarely arrive in an executable or verifiable form"	HDS6 — Tools, Repair, Alternatives, Coherence, Evidence, Scope — scored independently of final-task success
**This is the direct answer to the comparability problem this page has carried
since 2026-08-14.** Seven papers reporting harness-derived gains against five
mutually incomparable denominators (+17 points, +12.4%, +12.2 points, +3.4
points, and one ranking with no points at all) is not a bookkeeping annoyance —
Beyond Final Scores states the reason: **distinct process bottlenecks sit behind
similar final outcomes**. Two systems that score alike fail in different places,
so the score cannot attribute the gain to the harness, the model or the run.

Three findings from Beyond Final Scores bear directly on entries below it:

• "Engineering optimizers, not researchers." Agents formulate and implement practical solutions; their strongest solutions adapt or combine established techniques, and genuine methodological novelty remains rare.
• Run-to-run variance is substantial — reported as a finding rather than as noise to be averaged away.
• Experience reuse "can help or mislead" later decisions. Every self-improving loop on this page — DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545)'s archive of lineages, Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence (arXiv:2608.12743)'s scored lessons — assumes accumulation is monotone. Measured directly, it is not. Those results stand as things those loops achieved on their own benchmarks; they are not evidence of a general property.
• "Harness designs affect performance stability" — stated as one of the named causes, which is this page's thesis arriving as someone else's measured variable.

Apodex Discovery supplies the counter-weight, and it is the more interesting number here. The same environment lifts GPT-5.5 by 2.5 mean normalised prediction points and GPT-5.6-sol by 7.6 on the same biomedical task against the same closed-book backbone (source). If the harness were the capability, a 3× spread across two backbones would not appear. The honest reading of this page's own thesis, after today, is that the harness is a multiplier on the model, not a substitute for it — and nothing collected here so far could have distinguished those two claims.

Neither paper cites the other. The pairing is this wiki's and is labelled as such.

2026-08-17 — the memory slot, and the argument arriving from inside a 397B pre-training run

The five results above evolve prompts, tools, skills and control flow. Today's batch adds the slot they left out — what the model reads at inference — and, more significantly, one instance of the argument made by people who could have retrained instead (source):

Paper	What is evolved	Headline
Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence (arXiv:2608.12743)	retrieved experience, scored by its own transfer record	highest macro average in 4 of 4 base-model blocks, five spatial benchmarks
Intern-S2-Preview: Scientific Agentic Foundation Model (arXiv:2608.13505)	a 4B memory module beside a frozen 397B backbone	Biology-Instructions 56.92 → 60.32, backbone unmodified
SMA's contribution is that the memory measures itself. Each distilled lesson
carries a Transfer Reliability Score, initialised uniformly and then
calibrated from later retrieval outcomes, and retrieval ranks on similarity
and TRS together. Nothing else in this cluster has an internal component that
accumulates evidence about its own usefulness — and it is the exact instrument
Model Routing closed on as missing, in a different slot.

Intern-S2-Preview is the one that changes the argument's status. Every prior entry here is someone improving a system around a model they did not train. This is a 397B pre-training effort that allocated an architectural slot to specialising the model without modifying its own backbone — the Memory Decoder is described as "a separate memory-augmented path for rapid scientific specialization without modifying the frozen 397B backbone". When the people holding the weights choose the frozen-backbone path for a domain, "the harness is the capability" stops being a workaround for lacking a training budget.

The cost gap is unchanged and now spans seven papers. Not one of them publishes what a loop consumes — no compute cost, no population size, no retrieval overhead, no generation count. Seven independent groups agree the harness carries capability and none of them prices it.

And they still cannot be compared to each other. +17 points (DarwinX), +12.4% (AutoDesign), +12.2 points (SkillZip), +3.4 points on one domain average (Intern-MemDec), and a ranking with no points at all (SMA). Five different denominators. That is the state of the evidence, and it is the reason this page reports the cluster rather than a trend line.

2026-08-19 — the cluster's strongest result and its first real refutation land on the same day

Three papers in one HuggingFace batch, and for the first time they do not all point the same way (source):

Paper	Position	Headline evidence
StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089)	for	frozen runtime, weights untouched: GPT-5.6 Sol xhigh 95.3% on Terminal-Bench 2.1; DeepSeek-V4 Flash 82.7 → 88.1%; runbook transfers unchanged across a version bump
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905)	against	8 harness-model combinations × 100 tasks: the same 45 failure patterns recur across all 8, including the strongest models; deficit located at the model level
ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798)	neither	trains through the harness: Qwen3-30A3B gains +9.98 points via OpenClaw and +14.81 via Claude Code
The cost figure this page has asked for since 2026-08-17 finally exists. Seven
papers had reported harness-derived gains and none published what a loop costs.
StateM reports $15 of final-score API usage against $574.68 for the GPT
reference — a 38× ratio, at higher accuracy, on the same benchmark. That is
the first number here that lets a harness claim be priced rather than only ranked.

The refutation is the more important entry, because of how it is built. This page's standing complaint is that a result comes from one unnamed configuration, so a gain cannot be apportioned between model and scaffold. AutoResearchEval varies both axes and reports that the failure structure does not move. That is the ablation this cluster kept asking for, run against this cluster's own hypothesis, returning against it — and it says outright that whether orchestration-level interventions could close the gap is not tested.

The two are not in contradiction, and the joint reading is what this page now holds: harness scaling buys execution reliability and does not buy self-assessment. StateM's mechanism is durable state, checked transitions and recoverable runbooks — machinery for not losing track on specified, verifiable, bounded tasks. AutoResearchEval's missing faculty is a metacognitive loop: checking output against evidence, revising, questioning the path. No amount of state management supplies that, which is why all 8 combinations fail together. Neither paper cites the other; the pairing is this wiki's and is labelled as such.

ClawGym II makes the model/harness split harder to hold at all. Every other entry here treats the harness as a fixed scaffold around frozen weights. ClawGym II runs the arrow the other way — the harness becomes the environment the weights are trained against — and after that the model is no longer harness-agnostic. Its +9.98 vs +14.81 is also the cleanest measurement in this cluster of the harness as the only variable: same model, same benchmark, same method, 1.48× apart.

And a fourth component of "configuration" appears that no paper here names. Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (arXiv:2608.16391), from the same batch, audits what a vendor-hosted API actually serves and finds route-level fidelity loss that has little detectable association with GPQA-Diamond accuracy while coinciding with a declining Terminal-Bench pass rate as task exposure increases. Same benchmark family as StateM, same long-horizon regime. A reported score therefore now depends on the model, the harness, and the serving route — and single-turn knowledge benchmarks, which is what most third-party figures in this wiki are, are structurally blind to the third.

Standing count: ten papers in this cluster, still no shared denominator — and now one dollar figure, one negative result, and one that changes what the noun means.

2026-08-20 — the mechanism behind yesterday's joint reading, and the harness moves into training

Four papers in one HuggingFace batch, and between them they do something the cluster has not managed before: they say what the harness is doing, not only how much it is worth (source).

1. The channel is procedure, not knowledge. Demystifying Agent Skills: Why They Work — Until They Don't (arXiv:2608.14036) open-codes 8,135 trial records and finds procedural anchoring accounts for 65.7% of skill cases against 4.5% for explicit knowledge injection. Skills stabilise action; they do not supply missing facts. That is the same boundary this page recorded on 2026-08-19 — harness scaling buys execution reliability and does not buy self-assessment — arrived at by a third method, and it is now three papers agreeing without any of them citing another.

2. Retrieval quantity is not monotone in usefulness, and two papers say so from different sides. Skill retrieval actual-use precision falls 29.6% → 3.3% as the pool grows from 5 to 100; Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008) holds the harness fixed, varies only the memory substrate across 3 backbones and 4 suites, and reports that no substrate dominates — broad retrieval helps long-context factual QA and harms sequential decision-making by shifting attention off action-critical context. Every skill and memory figure this wiki holds is reported at one pool size, on one task class. Both of those are now known to be load-bearing.

3. The model/harness split is being dissolved from the training side. Agent Lightning v1.0: Towards Harnessed Agentic RL (arXiv:2608.17528) names the regime — harnessed agentic RL, where the deploy-time harness owns the interaction loop and the trainer sees only LLM request/response pairs — states that four other frameworks have adopted the architecture, and reports Qwen3.5-9B 41.8% → 56.4% on SWE-bench Verified from 6K examples. This is the mechanism behind ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798)'s awkward finding on 08-19 that a harness-trained model stops being harness-agnostic. If this is now the default post-training route, then this page's rule — a benchmark number is a claim about a (model, harness) pair — stops being a reporting convention and becomes a fact about the weights. A harness-agnostic model is something to demonstrate, not assume.

And the paper arguing hardest for the harness does not name its own. Agent Lightning's headline 56.4% is reported without stating which harness produced it — the exact omission this page exists to flag, in the paper whose thesis is that the harness participates in training. Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements (arXiv:2608.17310), landing the same day, makes the competing bet (evolution strategies instead of RL, to delete the credit-assignment machinery rather than fix it) and is not comparable to it: different model, benchmark, baseline and reporting convention, with no shared denominator. That is the sixth mutually incomparable denominator in this cluster.

4. Autonomy is a separate axis from harness quality, and it is where the collapse is. ASI-Bench: At the Dawn of Artificial Superintelligence (arXiv:2608.17271) holds the research project fixed and withdraws human methodological guidance in three steps: 50.91 → 29.10 → 26.62. Almost the entire loss lands on the first withdrawal, so what the guidance supplied was not mainly which method to pick. Read beside How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) — same deficit, different instrument — the boundary sharpens: agents execute research well when a human has already decided what the research is.

Practical consequence for this wiki

Benchmark rows on model pages record the figure with the harness or source that produced it where that is known, and a disputed figure goes to ## Conflicting Reports rather than being silently reconciled. The two "official harness" numbers for GPT-5.6 Sol — 7.8% verified by ARC Prize, 13.3% as re-run by OpenAI — are recorded as disagreeing because nothing read reconciles them.

The same discipline applies to the leaderboard snapshots this repo scrapes: LMArena is quoted as a percentage with its confidence interval rather than as Elo, and Artificial Analysis figures are quoted with the column heading above them (source).

Key Sources

• OpenAI, "How enabling two settings tripled our scores on the ARC-AGI-3 benchmark" (2026-07-29) (source)
• ARC Prize results — Claude Opus 5
• ARC Prize, analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3
• LMArena snapshot (source)
• Artificial Analysis snapshot (source)
• Qwen3.8-27B release — SWE-MM on the Claude Code harness (2026-08-14) (source)
• GLM-5.3 release — four benchmarks, no harness (2026-08-14) (source)
• AI4AI at Test-Time, arXiv:2608.12307 (source)
• HuggingFace Daily Papers, 2026-08-20 — Agent Skills, Harness the Memory, Agent Lightning v1.0, Agentic ESOpt and ASI-Bench (source)
• HuggingFace Daily Papers, 2026-08-19 — StateM, AutoResearchEval, ClawGym II and Ventor-QTest (source)
• HuggingFace Daily Papers, 2026-08-16 — DarwinX, AutoDesign, SHAPER and SkillZip (source)

Related Concepts

• Reasoning Models
• Test-Time Compute (Inference-Time Compute Scaling)
• Agents (LLM Agents)
• Conceptual Reasoning Index (CRI)
• Model Routing
• GPT-5.6 Sol (and Terra, Luna)
• Claude Opus 5
• DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545)
• StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089)
• How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905)
• ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798)
• Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (arXiv:2608.16391)