Repository-level coding benchmarks such as SWE-bench have become the standard for evaluating coding agents, but they carry an inherent flaw: they are built on popular open-source repositories that have been repeatedly used in model training. This raises a fundamental question—do strong scores reflect genuine repository reasoning, or memorization of familiar repository cues?
To investigate, researchers propose SchrodingerRepo (Schrödinger's Repository), an evaluation framework that treats the test repository as an evaluation-time latent variable. Rather than using a static representation, the repository is dynamically instantiated only when the agent enters the evaluation environment—much like Schrödinger's cat, whose final form is revealed only when the box is opened.
The instantiated repository preserves the original executable behavior while eroding familiar cues through four transformation levels:
- problem statement reconstruction,
- namespace remapping,
- intra-file layout reordering, and
- functionality-preserving code rewriting.
Evaluations on SWE-bench Verified and SWE-QA show that removing familiar repository cues consistently degrades agent performance and substantially increases interaction costs across models. Further analysis attributes the added cost primarily to greater difficulty in repository exploration and localization.
The findings suggest that current coding agents may partially rely on memorized repository-side cues, underscoring the need for evaluation under dynamically instantiated repository representations to better measure true code understanding.