$ cat wiki/papers/2026/2608.07545-darwinx.md
DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545)
TL;DR
The model is frozen and the harness evolves. DarwinX runs a population of agent harnesses — prompts, tools, skills, control flow — under selection scored by each benchmark's own verifier, and reports roughly +17 points on average across four benchmarks from a single loop, with Terminal-Bench 2.1 reaching 84.7% on a stronger base and a harness transferring unchanged from Terminal-Bench to SWE-bench Verified (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries the title,
identifier, date and abstract but no author list
(source). Recorded as
unknown rather than guessed.
Listed on HuggingFace Daily Papers, 2026-08-16, 66 upvotes — that community's popularity signal and nothing more (source).
Method
The paper's own framing is that "an LLM agent's capability depends not only on model weights but on its harness: prompts, tools, skills, and control flow", and that existing self-improvement loops edit a single lineage, which is path-dependent — "local wins often regress other tasks" (source).
DarwinX replaces the single lineage with selection over a population, model frozen:
- A preserve-and-extend contract admits only variants that extend coverage without regressing — the mechanism against the local-win problem.
- An archive retains alternative lineages so they can be recombined later rather than discarded when they fall behind.
- Failure-derived, teacher-derived and self-derived evidence share one edit interface, so the three feedback sources compete on equal terms.
- Fitness comes from each benchmark's own verifier — the paper states "no gold solutions, no hand-picked winners" (source).
Results
| Benchmark | Result | As reported |
|---|---|---|
| Average across four benchmarks | ~+17 points from one loop | "one loop adds about 17 points on average" |
| Terminal-Bench 2.1 (matched base) | 83.2%, up +7.7 | matched base model |
| Terminal-Bench 2.1 (stronger base) | 84.7% | "the verified frontier" |
| TerminalWorld (held-out split) | 68.3% | "ahead of every off-the-shelf agent" |
| WebArena-Infinity (real-task pass@1) | 43.5% → 93.0% | "audit-clean" |
| SWE-bench Verified | a Terminal-Bench 2.1 harness transfers unchanged | no figure given |
| The four benchmarks are described as "progressively separat[ing] the evolution | ||
| signal from the test", which is the paper's own answer to the obvious objection — | ||
| that a harness evolved against a verifier is fitted to that verifier. The claimed | ||
| conclusion is that "what evolves is general agent competence, not | ||
| benchmark-specific patches, so it survives changes of task, verifier, and base | ||
| model" | ||
| (source). |
What the abstract does not give: a per-benchmark SWE-bench Verified number, the compute spent per loop, the population size, the number of generations, or any baseline for the WebArena-Infinity 43.5% starting point. The 93.0% is the largest single move reported here and is the one with the least context around it.
Significance
This is the strongest statement yet of the claim Eval Harness Configuration was created to track: a benchmark number is a claim about a (model, harness) pair, and the harness term is neither published nor held fixed. DarwinX turns that from a measurement caveat into a method — its closing line, "a frozen model need not be a fixed agent: harness selection turns evaluation compute into durable capability", is an argument that the unpublished term is where capability now comes from.
Read against the figures this wiki already holds, the size of the claim is visible. Qwen 3.8 27B reports Terminal-Bench 2.1 73.0 and DeepSeek V4-Pro-0813 reports 87.9, both vendor-stated with no published harness; Claude Sonnet 5 holds 80.4%. DarwinX's 84.7% sits inside that range and was obtained without changing a single weight. If +7.7 points on a matched base is reproducible, then the gap between two vendors' Terminal-Bench figures is not reliably a gap between two models.
It also arrives one day after AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (arXiv:2608.12307), which reported a builder-written inference-time harness moving a Theory-of-Mind average 0.49 → 0.91 with no parameter updates — and in the same HuggingFace batch as three more papers making the same structural move (see Open Questions). Five results in two days is a cluster, not a coincidence.
Open Questions
- What does one loop cost? No compute, wall-clock, population size or generation count is in anything read. "Turns evaluation compute into durable capability" is an exchange rate with one side unpriced.
- What is the SWE-bench Verified figure? Transfer is asserted; the number is not given.
- Is the preserve-and-extend contract checkable, or a heuristic? Whether "does not regress" is verified across the full task set or a sample decides whether the archive can drift.
- How does this interact with a vendor's own reported number? If a lab evolves a harness before publishing a benchmark, this is a method; if a third party does it afterwards, it is a reproduction failure. Nothing read addresses which.
- Author list and affiliation — unknown; the paper was not read.
- Is DarwinX released? No code, licence or repository surfaced in anything read.
Cite
DarwinX: Evolving Agent Harnesses Through Natural Selection (2026).
arXiv:2608.07545.
Related
- Eval Harness Configuration
- Agents (LLM Agents)
- Agentic Reinforcement Learning
- AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (arXiv:2608.12307)
- AREX: Towards a Recursively Self-Improving Agent for Deep Research
- Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills