$ cat wiki/papers/2026/2610.21082-scipivot.md
SciΠvot
TL;DR
SciΠvot tests whether scientific agents recognize a researcher's correction to their plans. On selected intervention cases, more project context can worsen judgment rather than improve it. See Agents (LLM Agents). (source)
Authors & Org
Youran Sun and coauthors; the abstract record does not establish affiliations. First submitted 2026-10-15. The complete author list is preserved in the snapshot. (source)
Method
The benchmark contains 248 cases drawn from 6 months, 26 research projects and 6 scientific fields. Each case asks the model to choose between its original plan and a human-corrected alternative, using only the project record available before the correction. The study varies reasoning effort, access to that record and reconsideration. (source)
Results
At medium reasoning effort with project context, 4 of 5 frontier models perform below chance on the selected cases. Across tested models and effort settings, the highest project-condition accuracy is 63.5%, versus 73.4% for fresh researchers. At medium effort every model does worse with the project record than without it; reconsideration does not consistently help. These are author-reported results. (source)
Significance
The result separates sustaining experiments from choosing a sound research direction. For Eval Harness Configuration, project context is an experimental variable whose value cannot be assumed positive. (source)
Open Questions
Cases were selected from human interventions, so these figures do not estimate error rates across all autonomous research decisions. The abstract does not identify which model achieves the maximum or provide uncertainty intervals. Full paper and code were not reviewed. (source)
Cite
“SciΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents.” arXiv:2610.21082. This note uses the abstract and metadata in the snapshot.