$ cat wiki/papers/2026/2610.04184-evalresearchbench.md
EvalResearchBench
TL;DR
EvalResearchBench tests whether agents can build inexpensive evaluators that preserve target benchmarks' model rankings. The best evaluators agree on about 75% of candidate pairs, below the 91% ceiling defined by disagreements among the target benchmarks. These are ranking-agreement figures, not task-solving accuracy. (source)
Authors & Org
Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu and Huazheng Wang. First submitted 2026-10-03, revised 2026-10-16. The abstract page does not establish affiliations. (source)
Method
With target materials, development references, candidate APIs and fixed time/API budgets, a researcher agent selects or synthesizes tasks, implements graders and revises them through pilot tests. It then freezes an executable evaluator for coding, co-work and reasoning. The study compares 9 researchers, 13 candidate models and 14 target benchmarks, measuring score concordance and pairwise agreement. (source)
Results
No researcher leads on every metric. The best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost achieves the highest pairwise agreement. Pilot feedback repairs tasks and graders, but evaluators can still truncate answers, exhaust budgets or let a few questions dominate a domain score. (source)
Significance
Agents (LLM Agents) must evaluate the feedback instrument as well as the artifact it scores. Interpretation: Eval Harness Configuration needs held-out checks of ranking fidelity when an agent constructs its own cheap evaluator. The paper tests an ingredient of recursive self-improvement, not a demonstrated sustained self-improvement loop. (source)
Open Questions
The abstract does not give the absolute construction or execution costs, identify the winning evaluator, or establish whether ranking fidelity persists over repeated optimization against it. Full paper and code have not been reviewed here. (source)
Cite
arXiv:2610.04184 · Immutable abstract and submission-history capture of the revised version. (source)