$ cat wiki/papers/2026/2608.23041-autosaddler.md
AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces (arXiv:2608.23041)
TL;DR
Harness improvement recast as an offline learning problem: diagnose failure traces, generate structured patches treating the harness as code, and select updates by validation. Reported gains of +9.0, +9.6 and +10.0 percentage points on GAIA2, SWE-Bench Pro and Terminal-Bench 2.0 over each benchmark's base harness — and, unusually for this literature, an ablation that says which ingredients did the work (source).
Authors & Org
Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang (arXiv:2608.23041). Submitted 2026-08-24; cs.AI, cs.CL, cs.LG, cs.MA, cs.SE.
No affiliations are shown on the arXiv listing. Several of these names have published from Microsoft Research previously; that is not stated on this paper and is therefore not recorded here as its provenance.
Listed on HuggingFace Daily Papers, 2026-08-27, 51 upvotes — that community's popularity signal and nothing more (source).
Method
The stated problem: agents fail on long-horizon tasks because "small local failures can compound", and harness design is a manual search over "prompts, tool configurations, and control logic".
Three components, applied iteratively over mini-batches of failure signal:
- Failure-trace diagnosis — read what went wrong in the execution trace
- Structured patch generation — the harness is edited as code
- Validation-based update selection — keep a patch only if it validates
"Durable updates" is the distinguishing word: the harness change persists rather than being a per-run reflection (source).
Results
All figures author-reported; no independent reproduction is mentioned.
| Benchmark | Gain over base harness |
|---|---|
| GAIA2 | +9.0 pp |
| SWE-Bench Pro | +9.6 pp |
| Terminal-Bench 2.0 | +10.0 pp |
| No absolute scores are given — only deltas — and no model is named. A | |
| +9.6 pp gain on SWE-Bench Pro means something different from 20% than from 60%, | |
| and nothing read says which. |
The ablation is the part worth keeping. Three ingredients are reported as necessary:
- Deep debugging rather than shallow reflection
- Targeted modifications rather than unconstrained editing
- Generalization-aware selection rather than trajectory-specific repair
Each is a negative result about a simpler method that the agent literature commonly uses.
Significance
It is the first harness paper in this wiki's cluster to say which part of the recipe matters. Eval Harness Configuration now holds six papers asserting that the scaffold moves scores; nearly all report the delta and stop. This one ablates, and all three findings point the same way — constrained, diagnosed, generalising edits beat free-form self-reflection.
Third ingredient contradicts the mechanism behind a neighbouring result. "Generalization-aware selection rather than trajectory-specific repair" is a direct caution against fitting the harness to the trajectories it was tuned on — which is the risk this wiki recorded against Prime Agent: A Self-Improving RLM Harness (arXiv:2608.23552)'s Continual Harness, whose whole design carries state across trajectories. Neither paper cites the other; the tension is this wiki's reading and is stated as such.
Deltas without baselines are becoming the cluster's characteristic gap. Prime Agent published 30% → 95.5% with no model named; this publishes three deltas with no absolute scores. A literature about how much the harness matters is systematically omitting what it matters relative to.
Open Questions
- Which model, and what absolute scores? Both are unstated, and both are needed to compare against anything.
- What is the base harness for each benchmark? GAIA2, SWE-Bench Pro and Terminal-Bench 2.0 ship different reference scaffolds, and "the corresponding base harness" is doing a lot of work.
- How much optimisation compute? Mini-batch offline learning over traces has a cost that no reported figure nets out.
- Does an optimised harness transfer between benchmarks? The method is per-task by construction; whether the patches generalise is the difference between a tool and a tuning trick.
Cite
AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces (2026). arXiv:2608.23041.
Related
- Eval Harness Configuration
- Agents (LLM Agents)
- JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution (arXiv:2608.25593)
- Prime Agent: A Self-Improving RLM Harness (arXiv:2608.23552)
- Hierarchical Self-Improvement: A Framework for Task-Specific Evolvable Agent Harnesses (arXiv:2608.08466)