$ cat wiki/papers/2026/2611.20218-swe-cowork.md
SWE-CoWork — requirements through delivery
TL;DR
SWE-CoWork evaluates software delivery inside simulated organizations. High partial test coverage coexists with low complete-assignment success. (source)
Authors & Org
Xinping Lei and 25 coauthors; the full author list is retained in the snapshot. Affiliations are not established by the reviewed abstract record. Submitted 2026-11-15, v1. (source)
Method
The benchmark has 101 assignments across 101 real repositories, 11 programming languages and 12 application domains. Each simulated organization has 26 colleagues with distinct responsibilities, bounded knowledge and controlled collaboration. Agents start from a brief handoff, recover requirements from colleagues and records, reconcile conflicting evidence, and implement changes evaluated with private executable tests. 11 models are evaluated. (source)
Results
- The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. (source)
- Complete requirements upfront improve F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, with complete task success unchanged. (source)
- Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to accountable owners, yet obtains only 64% of colleague-held requirements. (source)
- Deferred questions go unfollowed; implementations remain incomplete after requirements are acquired; 31% of requirements reported complete fail their tests. (source)
Significance
Interpretation: Agents (LLM Agents) need sustained requirement tracking through clarification, implementation, verification and handoff. Eval Harness Configuration should distinguish partial credit from complete delivery; the two metrics answer different questions. (source)
Open Questions
This review covers the abstract and metadata, not full experiments or code. Transfer from simulated colleagues to real teams, model-specific uncertainty, and interventions that improve complete delivery remain unverified here. (source)
Cite
- arXiv:2611.20218, v1, 2026-11-15.
- Immutable abstract and metadata.