$ cat wiki/papers/2026/2608.15089-statem-harness-scaling.md
StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089)
TL;DR
An agent runtime that changes nothing about the model weights and reports 95.3% raw accuracy on Terminal-Bench 2.1 with GPT-5.6 Sol xhigh, against an 83.1% reference for GPT-5.5 xhigh. The same runtime carries a frozen profile across model families, lifting DeepSeek-V4 Flash from 82.7 to 88.1%. Its sharpest number is not accuracy: final-score API usage is about $15 against $574.68 for the GPT reference (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only. Code is stated at github.com/henryqin1997/statem, which was
also not read — github.com is under the same block
(source).
Listed on HuggingFace Daily Papers, 2026-08-19, 93 upvotes — that community's popularity signal and nothing more (source).
Method
The stated failure mode: long-horizon agents fail even when the underlying model can solve every constituent step — losing track of mutable state, failing to reactivate lessons from earlier executions, skipping known procedures, or stopping prematurely.
The bet is named explicitly as harness scaling: improve the execution system around an agent without changing its model weights. StateM is described as an agent-native runtime organised around:
| Element | What it is |
|---|---|
| Durable states | execution organised around persistent state rather than a transcript |
| Phase-local context | context scoped to the current phase |
| Checked transitions | state changes validated rather than assumed |
| Recoverable runbooks | procedures that survive and resume |
| Versioned procedural practices | practices agents and users can inspect together |
| The described mechanism is that selected postmortem findings become | |
| persistent, executable preconditions — learned controls made explicit and | |
| enforceable rather than left implicit in a prompt. |
Results
All figures as reported; none independently verified (source).
Terminal-Bench 2.1
| Configuration | Result |
|---|---|
| GPT-5.5 xhigh + StateM | 92.1% |
| GPT-5.5 xhigh reference | 83.1% |
| GPT-5.6 Sol Ultra (reference) | 91.9% |
| GPT-5.6 Sol xhigh + StateM | 95.3% raw accuracy across 445 trials; succeeds on all 89 tasks at least once |
| GPT-5.6 Luna, frozen profile | 76.7 → 85.4%, above the 84.9% Sol xhigh reference |
| DeepSeek-V4 Flash, adapted | 82.7 → 88.1% under standard timeouts; 89.1% on an 88-task common core |
| The runbook is stated to transfer unchanged from GPT-5.5 to GPT-5.6. The | |
| DeepSeek adaptation is stated to cost under $38. |
Cost
| Configuration | Final-score API usage |
|---|---|
| StateM | about $15 |
| GPT reference | $574.68 |
| Total DeepSeek expenditure | $52.22 |
| BusinessBench |
Family-specific runbooks built on development sets yield held-out gains of 0.55 macro and 1.34 micro points; two mechanism-matched families improve by 10.04 points. The paper's own reading: concrete rules generalise when tasks share execution structure, while the control methodology applies more broadly.
What the abstract does not give: the harness used for any reference figure, the 89-task and 88-task compositions, what "raw accuracy" excludes, the token counts behind the dollar figures, or a licence.
Significance
This is the strongest single datapoint Eval Harness Configuration has, and it arrives with the number that cluster has been missing for three weeks.
On the accuracy axis it argues the harness thesis about as hard as it can be argued. A frozen runtime moves four models from three families, transfers a runbook unchanged across a version bump, and puts a mid-tier model (DeepSeek V4-Flash) above where a frontier reference sat. The page's current, deliberately weakened position — a harness is a multiplier on the model, not the capability itself — survives this: every gain is still model-dependent (+9.0, +8.7, +5.4 points from the same runtime), and the 95.3% top line still needs the strongest available backbone underneath it.
On the cost axis it supplies what seven prior papers did not. The cluster's standing complaint, recorded since 2026-08-17, is that nothing publishes what a loop costs — seven papers, five mutually incomparable denominators, no denominator at all in dollars. StateM publishes $15 against $574.68, a 38× ratio, on the same benchmark at higher accuracy. That is the first figure in this wiki that lets a harness claim be priced.
It also lands on a live pricing argument. Test-Time Compute (Inference-Time Compute Scaling)
records three mechanisms that change the token count for the same answer without
changing the price per token, and the consequence that a Pricing row is a rate,
so none of them shows up in it. A 38× cost ratio at higher accuracy from a
runtime change is the same effect, at a magnitude no per-token price movement
this wiki has recorded comes close to.
The counterweight is same-day and is on the other side of the batch. How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) runs 8 harness-model combinations over 100 tasks and finds the failure patterns recur across all of them, including the strongest models, locating the deficit at the model level rather than in any scaffold. Neither paper cites the other; the pairing is this wiki's and is labelled as such. They are not straightforwardly contradictory — StateM is Terminal-Bench, AutoResearchEval is open-ended research — but the domains where harness scaling works and where it does not are now both on the record, on the same day, for the first time.
Open Questions
- What harness produced the reference figures? The 83.1% and 84.9% baselines carry no stated harness, which is the exact comparability defect Eval Harness Configuration exists to track. A harness-scaling paper reporting harness-free baselines is measuring against an unnamed configuration.
- What does "raw accuracy" exclude? The phrase appears in the title and is never defined in anything read; 95.3% raw over 445 trials alongside "succeeds on all 89 tasks at least once" invites reading two different quantities as one.
- Is the $15 comparable to the $574.68? One is "final-score API usage", the other a reference cost; whether development, adaptation and failed trials are inside either is not stated, and the $38 adaptation and $52.22 total for DeepSeek suggest they are not.
- Does the runbook transfer because the runtime generalises, or because both models are GPT? The unchanged GPT-5.5 → GPT-5.6 transfer is within one family; the DeepSeek case required paid adaptation.
- Author list, affiliation, licence — unknown; the paper was not read.
Cite
StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1
via Harness Scaling (2026). arXiv:2608.15089.
Related
- Eval Harness Configuration
- Test-Time Compute (Inference-Time Compute Scaling)
- Agents (LLM Agents)
- How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905)
- ClawGym II: Exploring Black-Box RL on Agent Harness (arXiv:2608.16798)
- DeepSeek V4-Flash
- GPT-5.6 Sol (and Terra, Luna)