EN한
← wiki

$ cat wiki/papers/2026/2611.15777-so-arena.md

SO-Arena

paperupdated 2026-11-15created 2026-11-15

TL;DR

Evaluate decision capability and reliability separately. Across 21 models, the authors find that the best-performing System One model ranks only seventh in reliability. (source)

Authors & Org

Lichen Li, Ziping Dong, Sicheng Feng, Qi Li and Xinchao Wang. Affiliations are not established by the captured metadata. First submitted 2026-11-11. (source)

Method

The benchmark spans 11 domains and five capability families, with manually verified reference labels, domain-specific difficulty criteria and interactive sandbox tasks. Reliability tests change option sets and ask whether the model recognizes when no valid answer is present. Reasoning-intensive tasks permit comparisons between System One and System Two approaches. (source)

Results

Scaling and extra reasoning help unevenly across domains. Strong aggregate task performance does not guarantee stable decisions or recognition of missing answers; the abstract does not identify the model behind its seventh-place reliability result. (source)

Significance

For Eval Harness Configuration, option construction and missing-answer handling are substantive evaluation dimensions. The paper complements Decision models as actors, judges and teachers, which distinguishes acting from judging and teaching; the studies do not share a common score. (source) (roles)

Open Questions

Full-paper methods, model-level tables and code were not reviewed. The abstract alone does not establish deployment reliability or statistical uncertainty around each ranking. (source)

Referenced by

Sources