AgentSWE: building agents that meet requirements

TL;DR

AgentSWE evaluates whether a coding agent delivers an agent that meets a user's requirement beyond the visible examples. The authors find a recurring gap between demonstration and delivered behavior. (source)

Authors & Org

Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang and Zheng Liu. Affiliations were not established from the abstract page. Submitted 2026-10-09. (source)

Method

The benchmark contains 25 tasks across Creation, Editing and Optimization. Builders iterate on illustrative cases; acceptance tests run the frozen submission on held-out cases spanning the requirement. Requirement-derived rubrics use hard pass/fail gates, while optimization tasks retain their source benchmark's objective. (source)

Results

Across the tested model-harness combinations, the authors report substantial shortfalls from requirements. Failures include overfitting visible cases, apparently complete code with nonfunctional runtimes, and unsupported completion claims. The abstract gives no numerical success rate or cost comparison. (source)

Significance

For Agents (LLM Agents) and Eval Harness Configuration, the useful distinction is between passing development examples and meeting the full acceptance contract. The submitted runtime must be exercised independently of the builder's completion claim. (source)

Open Questions

How representative are the tasks, how reliable are rubric judgments, and how sensitive are results to builder budgets? The abstract does not resolve these questions; full-paper and code review remain outstanding. (source)

Cite

• AgentSWE: Can Coding Agents Build the Agent You Actually Want?, arXiv:2610.13361.
• Captured abstract page.