Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

TL;DR

HoH is a framework that wraps existing coding-agent harnesses and organises their executions into iterative planning–coding–testing loops. Across three harness–model pairs on three benchmarks it reports an average relative gain of 52.25% and a maximum of 82.86% after three iterations over the standalone harness, and in a multi-day run of more than 70 iterations it autonomously develops a playable first-person-shooter game (source).

Authors & Org

Not published in anything read. The HuggingFace Daily Papers snapshot carries title, abstract and upvote count but no author list, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox, so no authors or affiliation are recorded rather than guessed. HuggingFace Daily Papers, 2026-09-03, 10 upvotes; arXiv publication date 2026-09-01 (source).

Method

HoH sits above a coding-agent harness rather than replacing one, which is where its name comes from. Its stated design commitments (source):

• balance repair against capability growth, rather than only fixing
• scope development into small, verifiable increments
• separate implementation-time testing from independent evaluation
• constrain verifiable outputs rather than prescribing agent workflows
• progressively expose deliverables, role-specific tools and skills
• encourage reuse rather than recreation
• maintain versioned project histories

The fourth is the load-bearing one. Constraining what must be true of the output and leaving the agent's procedure free is the opposite of the scaffold-heavy approach, and it is what makes HoH portable across harnesses it did not design.

Results

Three harness–model pairs, deliberately spanning vendors (source):

Harness	Model
Codex	GPT-5.5
OpenCode	DeepSeek-V4-Pro
Pi	MiniMax-M3
Benchmarks: GameCraft-Bench, FrontierSWE, ProgramBench. HoH "consistently
outperforms the corresponding standalone harnesses", **average relative gain
52.25%**, maximum 82.86%, after three iterations.

The multi-day deployment ran more than 70 iterations and produced a first-person-shooter game with "a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio".

Read the headline carefully: these are relative gains over each harness's own baseline, and no absolute score is given in anything read. A 52.25% relative improvement on a low base and on a high base are very different claims, and the abstract does not distinguish them. The game demonstration is likewise reported qualitatively — "human-playable", "polished" — with no human evaluation, rubric or count behind either word.

Significance

The result this wiki should take from it is not the percentage. It is that a framework layered over three different harnesses, on three different vendors' models, improved all three — which is another measurement of how much of a coding agent's reported capability lives in the scaffold rather than the weights. That is the through-line of Eval Harness Configuration, and this paper adds an unusually direct form of it: the model is held fixed and only the harness-above-the-harness changes.

It also pushes on a boundary Agents (LLM Agents) has been tracking. Most agentic coding evaluation is single-episode. HoH's unit is multi-day, 70+ iterations, with versioned history and continual improvement, where the failure modes are accumulation problems — dead code, structural erosion, verbosity — rather than task failure. Two other papers in the same week's intake attack the same class of problem from different angles: 2608.29310 Super Library Agent on redundancy across a portfolio of codebases, and EvoUndo: Recoverability-Constrained Self-Evolution for LLM Agent Harnesses on whether an agent's self-modifications can be recovered from.

Against AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling, captured the same day, there is a tension worth naming rather than resolving: HoH's improvements are scored on benchmarks, and AgentJudgeBench reports that judges of agentic work plateau at 77–82% on hard cases. Nothing read says how GameCraft-Bench, FrontierSWE or ProgramBench score their outputs, so whether HoH's 52.25% sits above or inside that band cannot be determined from either paper.

Open Questions

• No absolute scores, so the relative gains cannot be sited against any other system.
• "Three iterations" for the benchmark numbers versus "70+" for the demo. Where the improvement curve flattens — or whether it does — is unreported.
• Cost is absent. A framework that runs a harness in a loop for multiple days multiplies token spend, and no compute or dollar figure appears in anything read.
• The FPS game is a demonstration, not an evaluation. There is no baseline of what the same harnesses produce over 70 iterations without HoH.

Cite

arXiv 2609.01481, Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement, published 2026-09-01 — https://arxiv.org/abs/2609.01481 · code https://github.com/Flesymeb/HarnessOfHarness → (snapshot)