VeriHarness: checking agreement against evidence

TL;DR

VeriHarness gives a fixed model tools and a workspace to verify long-horizon agent outputs. It checks both disagreements and shared claims, because agreement can conceal mistakes. (source)

Authors & Org

The author line lists Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier, Tomas Pfister and Chen-Yu Lee. Affiliations are not established by the captured abstract page. Submitted 2026-10-01. (source)

Method

Repeated rollouts supply candidate artifacts. A disagreement resolver checks competing claims against environmental evidence; a consensus challenger tests shared claims and looks for omitted requirements. A workspace, evidence tools and reusable verification skills support selection and revision. No reference answers or grading rubrics are supplied at test time. (source)

Results

• Across five long-horizon workspace benchmarks and two models, the authors report the highest selection scores among tested baselines. The abstract does not provide the absolute scores. (source)
• Evidence-backed revision improves average performance over a single rollout by 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. (source)
• The authors release approximately 26,000 rollouts generated at a cost of over $100,000 and report that verification skills improve from failure feedback. That collection cost is not the marginal cost of deploying the verifier. (source)

Significance

For Agents (LLM Agents), consensus is a hypothesis to test against evidence rather than a completion certificate. This adds a verification mechanism to Eval Harness Configuration without changing model weights. The reported improvements remain results from the authors' evaluation. (source)

Open Questions

What is the extra verification cost per task, and how much of the gain survives a matched total-budget comparison? Neither is quantified in the abstract. Full-paper review is needed before interpreting this as a cost-efficiency result. (source)

Cite

arXiv:2610.00972. Read scope: abstract and author metadata, supplemented by the daily snapshot; the full paper was not read. (source)