$ cat wiki/papers/2026/2609.18805-programdistill.md
ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks
TL;DR
A coding-agent benchmark where the spec is a working application rather than an issue text: the agent has to discover what the software does by interacting with a reference app, then build it into an incomplete one. 4,063 tasks across 26 applications, built with no human intervention. GPT-6 Astra reaches 49.2% and Claude Opus 5 28.8% on cumulative workflows.
Authors & Org
Not stated. The HuggingFace Daily Papers snapshot carries the arXiv id, title,
upvote count, publication date and abstract; it carries no author list and no
affiliation, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox.
Recorded as unknown rather than guessed.
Method
Applications are factorized into features of different granularities, each
associated with replayable behaviors executable via its gold patch. The
pipeline, mine-craft-patch, discovers 1,975 replay-verified behaviors
across 26 applications and constructs 4,063 tasks without human
intervention
(source).
The premise the authors set against existing practice: coding agents are normally evaluated against desired behavior specified through issues or instructions, whereas in practical web development an agent may need to infer behavior from working software and implement it in an incomplete application. The reference app is the specification.
Difficulty is a dial, not a property of the task set — "restoration depth" controls how much of the target application has been removed.
Results
Nine frontier coding agents evaluated. Figures as published:
| Setting | Model | Result |
|---|---|---|
| Cumulative workflows, full-application reconstruction | GPT-6 Astra | 49.2% |
| Cumulative workflows, full-application reconstruction | Claude Opus 5 | 28.8% |
| Partial-application reconstruction, depth 1 → 8 | (first agent reported) | 100% → 64.0% |
| Partial-application reconstruction, depth 1 → 8 | (second agent reported) | 96% → 32% |
| The depth curves are the result worth keeping. Both agents start at or near | ||
| ceiling and one of them loses two thirds of its success by depth 8, the other | ||
| just over a third. A benchmark where the leaders sit at 100% and 96% at depth 1 | ||
| tells you nothing; the same benchmark at depth 8 separates them by **32 | ||
| points**. The abstract does not state which of the nine agents each depth | ||
| curve belongs to, so the two rows above are not attributed here. |
The abstract carries no aggregate score across all 4,063 tasks and no per-application breakdown.
Significance
This lands on the thread Eval Harness Configuration has been accumulating since W37: what the agent is told, and by what mechanism, is doing as much work as the model. ProgramDistill changes the specification channel itself — from prose to a running artefact the agent must probe — and the scores drop hard.
It is also the second measurement in two days on this wiki where GPT-6 Astra and Claude Opus 5 are separated by a wide margin on an agentic coding task, and the margin here is 20.4 points. This wiki holds no harness disclosure for either figure, which is exactly the gap that page exists to record.
The construction cost is the quieter claim: 4,063 tasks with no human intervention, verified by replay rather than by a human writing assertions. If that holds, the binding constraint on agentic coding benchmarks moves from annotation budget to whether you have applications to factorize.
Open Questions
- Which agents the depth curves describe. Two curves are published, nine agents were run, and the abstract attributes neither.
- What "cumulative workflows" scores against — the denominator is not stated, so 49.2% and 28.8% are comparable to each other and to nothing else.
- Whether replay verification admits a correct solution that differs from the gold patch. A behavior "executable via its gold patch" is a strong verifier and a narrow one; nothing in the abstract says how alternative implementations are judged.
- Whether the 26 applications are open — the abstract names no licence and no repository.
- Whether the harness was held constant across the nine agents. Not stated, and without it the 20.4-point gap is a comparison this wiki cannot check.
Related
- Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents — the other Paper Pick of 2026-09-21, and the complement to this one: ProgramDistill measures whether the agent succeeded, XConf measures whether it knew.
- Eval Harness Configuration — the thread this belongs to
- Agents (LLM Agents)