$ cat wiki/papers/2026/2608.14905-autoresearcheval.md
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905)
TL;DR
Runs 8 harness-model combinations over 100 research tasks, producing 800 annotated trajectories, and reports that the failure patterns recur across all 8 — including the strongest models tested. The paper's conclusion is therefore about where the deficit sits: at the model level, not in any particular scaffold. The named missing capability is a metacognitive loop (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only
(source).
Listed on HuggingFace Daily Papers, 2026-08-19, 22 upvotes — that community's popularity signal and nothing more (source).
Method
The subject is AutoResearch: a single system carrying whole-stage research from initial hypothesis to final published paper. The stated gap in existing evaluation is threefold — tasks are narrowly scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact-level visibility.
| Component | What it is |
|---|---|
| AutoResearchEval | 100 tasks grounded in published frontier science across 7 scientific domains, covering the full research lifecycle: ideation, retrieval, execution, analysis, writing, review |
| Trajectories | 8 harness-model combinations × 100 tasks = 800 trajectories, with process-level annotation |
| ARFT | AutoResearch Failure Taxonomy — 45 empirically grounded failure patterns |
| Attribution | a human-calibrated agent-as-a-judge pipeline inspecting complete trajectories and intermediate artifacts |
| Both AutoResearchEval and ARFT are stated to be publicly released. |
Results
The 45 failure patterns are stated to converge on a single overarching limitation: current agents lack a metacognitive loop, defined in the abstract as the ability to
- check what they produced against what they found,
- revise when it does not hold up, and
- question whether the path they took was sound.
The load-bearing result for this wiki: the same patterns recur across all 8 harness-model combinations, including the strongest models tested. The paper draws the conclusion explicitly — the deficit is at the model level rather than in any particular scaffold — and equally explicitly declines to close it: whether orchestration-level interventions can fix it is stated as an open question this work does not test.
What the abstract does not give: the 8 combinations, the 7 domains, any score or pass rate, the ARFT category structure, or the agreement rate of the agent-as-a-judge pipeline against its human calibration.
Significance
This is the first paper in the wiki's harness cluster whose primary finding is negative for the harness thesis, and it arrives on the same day as StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089), which is the strongest positive datapoint the cluster has. Neither cites the other; the pairing is this wiki's and is labelled as such.
The design is what makes it count. Eval Harness Configuration's standing complaint is that results are reported as one number from one unnamed configuration, so a gain cannot be apportioned between model and scaffold. AutoResearchEval varies both axes — 8 harness-model combinations — and reports that the failure structure does not move. That is the ablation the cluster has been asking for, run against the cluster's own hypothesis, and it comes back against it.
It does not refute StateM, and reading it that way would be wrong. The two measure different things: Terminal-Bench tasks are specified, verifiable and bounded, and StateM's mechanism is precisely durable state and recoverable procedure — the machinery for not losing track. AutoResearch tasks are open-ended, and the missing capability named here is judging your own output, which no amount of state management supplies. The defensible joint reading, and the one this wiki now records: harness scaling buys execution reliability and does not buy self-assessment. Where a task fails for the first reason a harness helps; where it fails for the second, all 8 combinations fail together.
It also sharpens a claim on Test-Time Compute (Inference-Time Compute Scaling). Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning (arXiv:2607.29211) argues that on beyond-capability inputs the correct budget is zero, naming systematic miscalibration and specious reasoning — superficially valid, subtly wrong. "Cannot check what it produced against what it found" is the same deficit observed from the evaluation side rather than the budget side. Two papers, different methods, same missing faculty; neither cites the other.
Open Questions
- Which 8 combinations? The whole model-level conclusion rests on the claim that the strongest models were included, and no model or harness is named in anything read.
- Is an agent-as-a-judge the right instrument here? How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (arXiv:2608.08975) shows an LLM evaluator's scores moving on rhetoric alone with reported content held fixed. A pipeline that judges research trajectories inherits that exposure, and "human-calibrated" is stated without an agreement figure.
- 45 patterns converging on one limitation — convergence on a single cause is a strong claim from a taxonomy the abstract does not describe; whether the patterns are independent or are 45 descriptions of one thing is not resolvable from anything read.
- Does the negative result survive a harness built for this? The paper says it does not test orchestration-level interventions. StateM's mechanism — postmortem findings turned into executable preconditions — is arguably an attempt at exactly the loop said to be missing, on a different task class.
- Author list, affiliation, licence — unknown; the paper was not read.
Cite
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100
Real-World Frontier Research Tasks (2026). arXiv:2608.14905.
Related
- Eval Harness Configuration
- Agents (LLM Agents)
- Test-Time Compute (Inference-Time Compute Scaling)
- StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089)
- Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning (arXiv:2607.29211)
- How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (arXiv:2608.08975)