RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

TL;DR

Measures the gap between how SWE-bench problems are written and how real users actually write requests, and finds it large: prompts carrying only a problem statement are 88% of real prompts but 7% of benchmark problems, and 87% of real prompts are casually written against 94% of benchmark problems being formal. Re-running seven models on realistic variants of the same tasks drops resolution rates by 6.4 pp on average and can change model rankings (source).

Authors & Org

Not published in anything read — the snapshot carries no author list and arxiv.org answers EGRESS_BLOCKED from this run's sandbox. HuggingFace Daily Papers, 2026-09-05, 16 upvotes; arXiv publication date 2026-08-31 (source).

Method

Two stages (source):

1. Characterise the gap. Define a six-category information taxonomy and four dimensions of linguistic style, then apply both to real user prompts from SWE-chat and to problem statements from SWE-bench Verified and SWE-bench Pro.

2. Build the controlled benchmark. RealSWE — 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within a family share the same underlying task and the same gold patch, and differ only in information composition and linguistic style.

That construction is what makes the result a measurement rather than a correlation: the code, the bug and the reference fix are held fixed, and the prompt is the only thing that moves.

Evaluation: seven contemporary LLMs.

Results

Finding	As reported
Problem-statement-only prompts	88% of real prompts · 7% of benchmark problems
Casual vs formal register	87% of real prompts casual · 94% of benchmark problems formal
Realistic inputs	resolution rates fall 6.4 pp on average
Ranking stability	realistic inputs can change model rankings
Which information matters	Desired Behavior and Motivation significantly affect performance
Which does not	Environment Information and Reproduction Steps "merely add tokens without measurable benefit"
Linguistic style	small, model-dependent effects only
The style result is the one worth not skipping. The intuitive reading of this
paper — real users write sloppily, so agents do worse — is the reading the paper
rejects. Register barely matters; **what matters is which of the six information
categories the prompt contains**, and the two that matter are the two real
prompts most often omit.

Significance

Eval Harness Configuration has argued since 2026-07-31 that a benchmark number is a claim about a (model, harness) pair. This extends the pair to include the prompt distribution — and does it with the ranking result, which is the part that has teeth. A configuration difference that moves every model by the same amount is a calibration issue; one that reorders models means two labs quoting SWE-bench at each other may be comparing artefacts of prompt style.

That page already holds ARC-AGI-3 spanning 7.8% → 38.3% on GPT-5.6 Sol (and Terra, Luna) from harness settings alone, and — from 2026-09-04 — Astra's ARC-AGI-3 carried at both 98.6% and 99.9% with the higher figure conditioned on OpenAI's own provider-adapter harness. RealSWE adds a 6.4 pp effect from a variable nobody reports at all. It is smaller than either, and it is the one that applies to every SWE-bench figure this wiki holds — including Claude Opus 5's 96% SWE-bench Verified, Claude Fable 5's 80.3% SWE-bench Pro, and the 09-03 finding on Gemini 3.8 Flash that Terminal-Bench 2.1 moved 9.2 points against SWE-bench Pro's 1.2.

It does not invalidate those numbers and this page does not claim it does. All of them were measured on the formal distribution, consistently; the finding is that the distribution is not the deployment one, and that the ordering is not guaranteed to survive the change.

There is also a practical result here that is unusual for an eval paper and worth stating plainly: telling an agent the desired behaviour and the motivation is worth more than giving it reproduction steps and environment detail. That is actionable for anyone using a coding agent, independent of the benchmark argument.

Open Questions

• Which seven models? Unnamed in anything read, so the ranking-change claim cannot be checked against any pair this wiki tracks — which is precisely the claim that matters
• How large is the reordering? "Can change model rankings" does not say whether it reorders adjacent models or distant ones. Adjacent swaps at a 1-point gap would be unremarkable; the 09-03 note on Qwen 3.8 Max — a three-point claimed margin against a ±18 interval — is the standing example of why the distinction matters
• Is SWE-chat representative of "real"? One prompt corpus is standing in for all real user behaviour; nothing read describes its provenance or scale
• Do the gold patches still fit? Variants share the gold patch by construction, but a prompt that omits the desired behaviour may admit several correct fixes. Grading against one patch would then score prompt-inference, not repair — and would inflate the reported drop
• Does the 6.4 pp gap close with scaffolding? An agent that asks a clarifying question is the obvious remedy, and nothing read tests one

Cite

RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests. arXiv:2608.27831, 2026-08-31. Recorded from HuggingFace Daily Papers, 2026-09-05 (source).