$ cat wiki/papers/2026/2610.35138-co-design-bench.md
Co-Design Bench — completing every requested revision
TL;DR
Co-Design Bench evaluates application revisions against hidden acceptance criteria using source and browser evidence. Its strongest tested endpoint satisfies many individual criteria but completes far fewer whole tasks. (source)
Authors & Org
Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang, Weixiang Yu, Cheng Lu, Chonggao Sun, Tong Zhou, Dahua Lin, Hang Zhang, Shuai Bai and Dayiheng Liu. Affiliations are not established in the captured abstract record. First submitted 2026-10-26. (source)
Method
The benchmark contains 273 tasks, curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Multimodal Editing (ME) covers localized changes with optional image, audio, video or document inputs; Repository Editing (RE) covers coordinated repository revisions. The three capability dimensions are grounded understanding, UI transformation and functional implementation. (source)
Normalized requests and runnable pre-edit applications are paired with 2,331 hidden expert-authored acceptance criteria. A judge grades criteria from source and browser evidence, a separate LLM verifier challenges positive verdicts, and visual and interaction claims are audited without gold code or target screenshots. (source)
Results
Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task, yet fully satisfies only 26.4% of tasks. All ten endpoints have lower criterion completion on repository edits involving three or four change types than on edits involving one or two. These are different outcome measures, not two estimates of the same success rate. (source)
Significance
For Agents (LLM Agents) and Eval Harness Configuration, the useful distinction is partial requirement satisfaction versus complete delivery. Interpretation: retaining both measures makes remaining work visible when an average score looks strong. The separate verifier also tests positive judgments rather than accepting them unchallenged. (source)
Open Questions
The abstract does not name the leading endpoint, quantify judge error or give inference costs. Generalization beyond this curated task set and independent reproduction remain unestablished in this review. Full-paper details and code were not reviewed. (source)