$ cat wiki/papers/2026/2610.24649-good-judgment-bad-advice.md
Good Judgment, Bad Advice
TL;DR
A pre-specified terminal-agent study finds that Jev 1.13 improves offline decision ranking but increases cost without detectable success gains. Knowing when an agent is finished did not translate into useful interventions in this configuration. (source)
Authors & Org
Naman Vats. Affiliation is not established by the abstract record. First submitted 2026-10-17; captured 2026-10-22. (source)
Method
On Terminal-Bench 2.1, the study fixes a Claude Sonnet 5.5 agent loop and varies its controller: none, deterministic rules, Claude Haiku 5.5, Jev 1.13, or a placebo sampling Jev's development-phase choice frequencies at the same decision points. The held-out evaluation covers 21 tasks and 315 trials. Full paper and code were not reviewed. (source)
Results
Jev raises cost per attempt by 19%: geometric-mean ratio 1.19, 95% CI 1.02-1.36, with identical 73.0% vs. 73.0% pass rates. It is cheaper than the small-LLM controller, ratio 0.81, but 22% more expensive than the placebo; no pass-rate difference between arms is detectable. (source)
Offline finish-request ranking reaches AUROC 0.78, versus 0.62 for rules and 0.60 for the small LLM. However, Jev's mean completion probability is 0.50 when 87% of those runs pass, prompting verification of already-correct work. No controller ever chooses to abandon a run. These are author-reported results for this setup. (source)
Significance
The study separates predictive ranking, calibration and intervention value. Model Routing needs end-to-end cost and success measurements, while Eval Harness Configuration gains a placebo control for whether decision timing and frequency alone explain apparent benefits. (source)
Open Questions
The abstract does not establish whether recalibration, other intervention rules or other workloads reverse the result. Failure to detect a pass-rate difference on this task set is not proof of equivalence across deployments. The results concern Jev 1.13 in this harness, not every decision model or routing system. (source)