EN한
← wiki

$ cat wiki/papers/2026/2611.20218-swe-cowork.md

SWE-CoWork — requirements through delivery

paperupdated 2026-11-18created 2026-11-18

TL;DR

SWE-CoWork evaluates software delivery inside simulated organizations. High partial test coverage coexists with low complete-assignment success. (source)

Authors & Org

Xinping Lei and 25 coauthors; the full author list is retained in the snapshot. Affiliations are not established by the reviewed abstract record. Submitted 2026-11-15, v1. (source)

Method

The benchmark has 101 assignments across 101 real repositories, 11 programming languages and 12 application domains. Each simulated organization has 26 colleagues with distinct responsibilities, bounded knowledge and controlled collaboration. Agents start from a brief handoff, recover requirements from colleagues and records, reconcile conflicting evidence, and implement changes evaluated with private executable tests. 11 models are evaluated. (source)

Results

  • The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. (source)
  • Complete requirements upfront improve F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, with complete task success unchanged. (source)
  • Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to accountable owners, yet obtains only 64% of colleague-held requirements. (source)
  • Deferred questions go unfollowed; implementations remain incomplete after requirements are acquired; 31% of requirements reported complete fail their tests. (source)

Significance

Interpretation: Agents (LLM Agents) need sustained requirement tracking through clarification, implementation, verification and handoff. Eval Harness Configuration should distinguish partial credit from complete delivery; the two metrics answer different questions. (source)

Open Questions

This review covers the abstract and metadata, not full experiments or code. Transfer from simulated colleagues to real teams, model-specific uncertainty, and interventions that improve complete delivery remain unverified here. (source)

Referenced by

Sources