Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?

TL;DR

Proposes SchrodingerRepo, which rebuilds the test repository at evaluation time so the agent meets a codebase that behaves identically but looks unfamiliar. On SWE-bench Verified and SWE-QA, removing familiar repository cues consistently degrades performance and substantially increases interaction cost across models — the extra cost falling on exploration and localization (source).

Authors & Org

Not stated in the snapshot — no author list, no affiliation. arxiv.org answers EGRESS_BLOCKED from this run's sandbox. Recorded as unknown rather than guessed.

Method

The stated premise: repository-level coding benchmarks are built on popular open-source repositories repeatedly used for training, so they inherently suffer from data leakage, and strong scores may reflect memorization of canonical repository cues rather than robust repository reasoning.

SchrodingerRepo treats the test repository as an evaluation-time latent variable, instantiated only when the agent enters the evaluation environment, instead of being a fixed artifact the agent may already have seen. Each instantiation preserves the original executable behavior while eroding familiarity through four transformation levels:

1. Problem statement reconstruction
2. Namespace remapping
3. Intra-file layout reordering
4. Functionality-preserving code rewriting

Evaluated on SWE-bench Verified and SWE-QA.

Results

Claim	Reported
Effect of removing familiar cues	consistently degrades agent performance across models
Effect on cost	substantially increases interaction costs
Where the cost goes	repository exploration and localization become harder
Conclusion drawn	current coding agents may partially rely on memorized repository-side cues
Every result in the snapshot is directional. There is **no score, no
percentage-point drop, no cost multiplier and no model name** anywhere in the
abstract — "consistently degrades" and "substantially increases" are the whole
quantitative record available here.

Significance

This is the contamination question asked as an experiment rather than as a statistic, and it lands on the benchmark this wiki has been reading model launches against all year. The usual contamination check compares a benchmark's scores against a held-out or post-cutoff set; SchrodingerRepo instead keeps the task and removes the familiarity, so the difference is attributable to cues rather than to difficulty.

It matters here for a specific reason. Two of the last three weeks' vendor launch tables were argued over on exactly this ground. Claude Opus 5.5 shipped on 2026-09-22 with no SWE-bench-family row at all, and GPT-6 Sol and GPT-6 Luna shipped the same day with DeepSWE v1.1 as their only benchmark. This paper supplies an argument for why a vendor might move off SWE-bench that is not "our model does worse on it" — and it is the first such argument this wiki holds that is testable.

The other half belongs to Eval Harness Configuration: an agent that gets more expensive when the repository is unfamiliar is an agent whose reported cost figures are a function of which repository it was measured on. Every per-task cost claim recorded on this wiki — Sol at "~80% below Fable 5", Luna at "93% below Claude Opus 5" — was measured on canonical repositories. Whether those ratios survive an unfamiliar codebase is not established by anything, including this paper.

The link to The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks is the same instrumentation turn from a different side: Taste-Bench asked whether the trajectory's decisions are being measured, this asks whether the environment is.

Open Questions

• How large is the degradation? No figure of any kind. Without one, "current coding agents may partially rely on memorized cues" is unfalsifiable as stated.
• Which models were evaluated? "Popular LLMs" is the whole identification.
• Do the four transformation levels degrade independently? Namespace remapping and code rewriting are very different interventions, and no ablation is reported in the snapshot.
• Does "functionality-preserving code rewriting" preserve difficulty? A rewrite that makes code harder to read for reasons unrelated to memorization would produce the same result.
• What harness ran the agents? Unnamed — the standing gap Eval Harness Configuration exists to record.
• Is SchrodingerRepo released? No repository, licence or artifact is named in the snapshot.

Cite

arXiv 2609.27891, Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?, HuggingFace Daily Papers 2026-09-25, 14 upvotes (snapshot).