HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

TL;DR

A benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure: the model is scored on the harness it builds, not on the tasks it solves. Generated harnesses stay substantially behind mature human-engineered references on code and on search/research, while matching or exceeding them on writing and machine-learning experimentation. The Evolution stage produces gains that are unstable and transfer only partially to held-out tasks (source).

Authors & Org

Not published in anything read. The HuggingFace Daily Papers snapshot carries title, abstract and upvote count but no author list, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox. HuggingFace Daily Papers, 2026-09-04, 219 upvotes — the second-highest entry in that snapshot; arXiv publication date 2026-09-01 (source).

Method

Two stages, both with model weights held fixed (source):

• Creation — the agent starts from a minimal seed and a small number of cases and builds a complete execution system
• Evolution — it starts from its own created harness and iteratively revises it using downstream execution feedback, aiming to improve benchmark performance

Each constructed harness is then evaluated on two axes: capability (task success on held-out benchmarks) and efficiency (execution-token cost). Reported Creation results cover six creator LLMs, four domains and five downstream benchmarks totalling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development.

The premise is stated in the abstract and is the part this wiki has been accumulating independently: changing the harness while holding weights fixed can substantially alter task performance, and current agent evaluations report downstream performance under a chosen harness, leaving the ability to develop the harness itself unexamined.

Results

Finding	As reported
Code, search and research	generated harnesses substantially behind mature human-engineered references
Writing, ML experimentation	matching or exceeding the selected references
Execution cost	large variation across generated harnesses
Evolution stage	gains are produced, but unstable, and transfer only partially to held-out tasks
Fixed-runtime-model experiment	gains depend strongly on the model executing the harness — limited transfer across models
The last row is the one with teeth. A harness that improves results only when run
by the model that wrote it is not infrastructure; it is a model-specific artefact
that happens to be written in code.

Significance

Three papers in one snapshot make the harness the object rather than the setting, and none of them cites the others. HarnessDev builds and evolves one; Aspire: Can Models Self-Evolve from Vague Goals? evolves both weights and harness from a vague goal; Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills holds the harness fixed and moves the operational knowledge out of it. Filed together on Eval Harness Configuration as this wiki's reading, not as a claim any of them makes.

Against what that page already holds:

• LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering (2026-09-01) made the Controller the subject of evaluation. HarnessDev goes one step further and makes the construction of the controller the subject
• Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement (2026-09-03) reports +52.25% average relative by wrapping existing harnesses in an improvement loop. HarnessDev reports that harnesses generated from scratch stay behind human-engineered ones on the domains HoH tests. These are not in contradiction — one improves a mature harness, the other builds one from a seed — but they are the two halves of the same question and they answer it in opposite tones
• EvoUndo: Recoverability-Constrained Self-Evolution for LLM Agent Harnesses (2026-09-02) treats the harness as a mutable object and finds conventional repair recovers 0/197 capability mutations. HarnessDev's unstable Evolution gains are the same instability measured from the performance side rather than the recoverability side

Open Questions

• Which harnesses were the human-engineered references? The abstract names none. Without them, "substantially behind" has no scale, and the writing and ML-experimentation wins have no baseline to be wins over
• Is the code/writing split about the domain or about the reference? Mature open coding harnesses have had years of engineering; a mature writing harness may simply not exist to lose to. Nothing read distinguishes "models are worse at building code harnesses" from "code harnesses are the ones humans already built well"
• What does the efficiency axis conclude? "Large variation in execution cost" is reported as an observation, and no cost-adjusted ranking is given
• The cross-model transfer result is the most consequential and the least detailed. If a generated harness's gains depend on the executing model, the Evolution stage may be fitting the harness to a model rather than to a task — which would make it a form of prompt optimisation with a filesystem

Cite

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? arXiv:2609.01437, 2026-09-01. Recorded from HuggingFace Daily Papers, 2026-09-04 (source).