$ cat wiki/papers/2026/2609.20804-harness-design-coding-agents.md
An Empirical Study of Harness Design for Coding Agents
TL;DR
Holds a coding agent's execution loop fixed and varies three components — planning, action space, context management — across 176 matched settings and four models, and reports that which component helps depends on the model and the context budget, not on the component (source).
Authors & Org
Not stated. The HuggingFace Daily Papers snapshot carries the arXiv id, title, upvote count, publication date and abstract; it carries no author list and no affiliation, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox. Recorded as unknown rather than guessed.
Published 2026-09-17, 37 upvotes in the 2026-09-19 snapshot — that community's popularity signal and nothing more (source).
Method
A lightweight coding harness whose execution loop is fixed while three components vary, which is what makes the comparisons matched rather than system-against-system (source):
| Varied | Range |
|---|---|
| Context management | five strategies |
| Context-window budget | four budgets |
| Planning | targeted ablations |
| Action space | targeted ablations |
| 176 matched settings, four models, evaluated on SWE-Bench Verified and Terminal-Bench 2.1. The stated motivation is that existing work "typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear" (source). |
Results
Four findings, stated as the paper states them (source):
- Context management earns its keep as the budget tightens, and "most of its benefit comes from preventing context-overflow failures" — a failure mode, not a reasoning gain.
- Staging rule-based elision before LLM-based summarization is the strongest strategy. Making elided content recoverable "adds machinery that models rarely use and yields no accuracy gain."
- Planning changes role with model strength: "an accuracy scaffold for weaker models" becomes "a cost saver for stronger models, with little change in accuracy."
- Predefined tools help models with weaker bash proficiency; bash-capable models "can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks."
Trajectory-level analysis is offered as the mechanism: context management extends trajectories without substantially altering behaviour, planning changes where trajectories stop, and the action space changes the granularity at which code is written (source).
No absolute score appears in the abstract, which is the only text this wiki holds — no per-model table, no baseline number, and no named models among the four.
Significance
It is the first paper on this wiki that treats the harness as something with parts. Eval Harness Configuration exists because benchmark figures on model pages are produced by unnamed harnesses and are therefore not comparable; this measures what the unnamed thing is actually doing, component by component, and finds that two of the three components do different jobs for different models. A harness is not a constant being held equal across a comparison — it is an intervention whose effect has the model's strength in it.
Finding 3 is the one that lands on this wiki's existing material. If planning is an accuracy scaffold for weak models and a cost saver for strong ones, then a harness tuned on one model class is mis-tuned for the other, and the 2026-09-18 Databricks account — engineers' coding spend up ~60% after moving to a stronger model — is exactly the shape of a cost that a harness re-tuning is supposed to address and nothing read says was attempted.
Finding 2 is a negative result with a design consequence. Recoverable elision is the more principled design — nothing is destroyed — and it "yields no accuracy gain". That is worth more than the positive findings, because it is the one nobody would publish if the field rewarded only improvements.
Read alongside 2609.20519 (SoL-Pi), which is in the same snapshot and reaches a compatible conclusion from the other direction: SoL-Pi discovers four harness mechanisms automatically — action execution, context compaction, observation handling, delegated reading — and reports 44.7–49.0% less token traffic at comparable performance on the 51-task EdgeBench (source). Three harness papers in one snapshot — this one, SoL-Pi and Agora: Git as Shared Memory for Collective AutoResearch — is this wiki's observation, asserted by none of them.
Open Questions
- Which four models. The abstract names none, so "planning is a cost saver for stronger models" has no boundary this wiki can state.
- What the five context-management strategies are, beyond the two named in finding 2.
- Whether the ordering survives a harness whose execution loop is not fixed — the fixed loop is what makes the study matched and also what makes it about this harness.
- No cost figures. "Substantially lower cost" and "cost saver" appear without dollars or tokens, where SoL-Pi in the same snapshot publishes both.
- Whether the context-overflow finding is a fact about context management or about the four budgets chosen.
Cite
arXiv 2609.20804 — An Empirical Study of Harness Design for Coding Agents, 2026-09-17. Captured from HuggingFace Daily Papers, 2026-09-19, 37 upvotes.