$ cat wiki/papers/2026/2608.13417-beyond-final-scores.md
Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417)
TL;DR
Evaluates seven frontier models on 36 long-horizon AI-R&D tasks with rule-based process metrics instead of final scores alone, and reports that current agents "operate more like engineering optimizers than fully autonomous researchers": they implement practical solutions, but run-to-run variance is substantial, their strongest solutions adapt or combine established techniques, and genuine methodological novelty is rare. Among the named causes of the observed performance: harness designs that affect performance stability (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only. No lab, author or affiliation is named in anything read
(source).
Listed on HuggingFace Daily Papers, 2026-08-18, 41 upvotes — that community's popularity signal and nothing more (source).
Method
The stated objection to score-only evaluation is twofold: a final score neither reveals where progress is gained or lost, nor whether accumulated experience improves later decisions.
The framework has two halves (source):
| Half | What it measures |
|---|---|
| Rule-based within-run metrics | behaviour along three axes — Solution Framing, Execution, Feedback Control |
| Controlled comparisons | experience reuse within a task and across tasks |
| Scale | As reported |
| --- | --- |
| Models evaluated | 7 frontier models |
| Tasks | 36 long-horizon |
| The three within-run axes are the paper's own names and are the load-bearing | |
| construct; the abstract does not define the rules behind them. |
Results
Stated findings (source):
- Agents "can formulate and implement practical solutions" — the engineering half works.
- Performance varies substantially across runs. Variance is a finding, not noise to be averaged out.
- Strongest solutions mainly adapt or combine established techniques; genuine methodological novelty remains rare.
- Distinct process bottlenecks sit behind similar final outcomes — two agents can score the same and fail in different places.
- Experience reuse can help or mislead subsequent decisions. Not monotone.
- Harness designs affect performance stability.
Four directions are named as consequences: model training, inference-time strategies, experience management, and harness design.
No numbers of any kind appear in the abstract — not a score, not a variance, not a per-axis breakdown, and the seven models are not named. Every result above is qualitative, and this page states them as the paper's claims rather than as measurements.
Significance
This is the first paper in the Eval Harness Configuration cluster to attack the measurement, rather than proposing another harness. The cluster's running complaint — recorded on that page since 2026-08-14 — is that seven papers report harness-derived gains against five mutually incomparable denominators (+17 points, +12.4%, +12.2 points, +3.4 points, and one ranking with no points at all). This paper says the denominator was the wrong object: a final score cannot attribute a gain to the harness, the model or the run, and "distinct process bottlenecks behind similar final outcomes" is exactly the failure that makes those five figures uncomparable.
It cuts against the cluster's thesis in one specific place. DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545) reports ~+17 points average from evolving harnesses over a frozen model, and Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence (arXiv:2608.12743) evolves what the model reads with a self-calibrating reliability score. Both depend on experience accumulating usefully. This paper measures that directly and finds reuse "can help or mislead" — so the accumulation those methods assume is not established as a general property, only as an outcome those particular loops achieved on their own benchmarks.
"Engineering optimizers, not researchers" is also the sharpest statement yet of where Agents (LLM Agents) actually is on long-horizon work, and it is stated from process evidence rather than from a leaderboard.
Open Questions
- Which seven models? An unnamed model set makes every claim unfalsifiable against a specific system, and "frontier" was a moving set across the eight weeks this paper's tasks would have been run in.
- What are the rules? Solution Framing, Execution and Feedback Control are measured "rule-based"; if the rules are hand-written, they are a harness of their own, with the same attribution problem the paper raises.
- How much variance? "Substantial" without a figure cannot be compared to any of the harness papers' reported gains — which is the comparison the paper's own argument demands.
- Is novelty measured or judged? "Genuine methodological novelty remains rare" is the strongest claim here and the abstract gives no instrument for it.
- Author list, affiliation, code and task availability — unknown; the paper was not read.
Cite
Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI
Research and Development (2026). arXiv:2608.13417.
Related
- Eval Harness Configuration
- Agents (LLM Agents)
- Agentic Reinforcement Learning
- DarwinX: Evolving Agent Harnesses Through Natural Selection (arXiv:2608.07545)
- Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence (arXiv:2608.12743)
- Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence (arXiv:2608.11341)