$ cat wiki/papers/2026/2608.19799-swe-bench-science.md
SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? (arXiv:2608.19799)
TL;DR
A repository-level benchmark of 119 tasks from 98 GitHub repositories across 20 scientific domains, in three paradigms (Issue-driven, Expert-exploratory, Engineering-integration). The best agent — Claude Code with Opus-5 (max) — scores pass@1 below 50%. A paired ablation finds scientific knowledge is not uniformly beneficial: well-aligned guidance helps and saves tokens, poorly aligned guidance anchors the agent and does not (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED; the paper was not read. Listed
on HuggingFace Daily Papers, 2026-08-22, 51 upvotes — that community's popularity
signal only (source).
Method
Premise: software is part of the scientific instrument, so a bug in scientific code can compromise not just program behavior but the evidence underlying a conclusion. Existing coding-agent evaluations emphasize aggregate success and give little insight into why agents fail on scientific software.
- 119 tasks / 98 repositories / 20 scientific domains.
- Three paradigms: Issue-driven, Expert-exploratory, Engineering-integration.
- A paired ablation removes explicit scientific guidance while preserving the repository and executable engineering context.
Results
- Best agent Claude Code + Opus-5 (max): pass@1 < 50%.
- Four recurring failure mechanisms: (1) deficits in scientific knowledge or abstraction; (2) misguided exploration / surface-level repair; (3) incomplete repair coverage or system integration; (4) failure to generalize scientific knowledge beyond observed cases.
- Ablation: scientific knowledge is not uniformly beneficial — well-grounded information can constrain repair, improve average performance and token efficiency, whereas poorly aligned guidance induces anchoring and does not necessarily improve exact repair success.
What the abstract does not give: the full model leaderboard, per-paradigm scores, and how "aligned" vs "poorly aligned" guidance was operationalized.
Significance
Two things land at once. First, a frontier ceiling: the strongest publicly named coding agent this wiki tracks — Claude Code on Claude Opus 5, whose page records SWE-bench Verified 96% — clears under 50% here. That gap is the benchmark's whole argument: repository-level scientific engineering is a different and harder distribution than the SWE-bench Verified tasks that saturate.
Second, the ablation is a clean Eval Harness Configuration result from the information side rather than the harness-settings side: adding domain knowledge is not monotone in usefulness, and mis-aligned guidance actively anchors the agent. That mirrors Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008) (more retrieval is not more useful; the task class decides the sign) and MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use (arXiv:2608.20202) (relevant memories can distort reasoning) — three instruments this fortnight reporting that more context is not better context.
Caveat kept: a single named result (Opus-5 max) is the only score in the abstract, so the "< 50%" ceiling is one data point, and the model roster and its spread are not readable here.
Open Questions
- What is the full model leaderboard, and the spread across the three paradigms? Unstated beyond the single Opus-5 figure.
- How is guidance alignment measured? The anchoring result depends on that definition.
- Does the failure taxonomy point to a fixable harness change, or a model-level scientific-knowledge deficit like How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) found?
- Author list, affiliation, licence — unknown; the paper was not read.
Cite
SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science? (2026). arXiv:2608.19799.
Related
- Agents (LLM Agents)
- Eval Harness Configuration
- Claude Opus 5
- Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008)
- MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use (arXiv:2608.20202)
- How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905)