Software is increasingly part of the scientific instrument itself, meaning bugs in scientific code can undermine not just program behavior but the evidence behind scientific conclusions. Yet existing evaluations of coding agents mostly focus on aggregate task success, offering little insight into why agents fail when repairing scientific software.
To address this, researchers introduce SWE-bench Science, a repository-level benchmark for scientific software engineering. It comprises 119 tasks from 98 GitHub repositories across 20 scientific domains, organized into three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration.
Even the best-performing agent, Claude Code with Opus-5 (max), achieves a pass@1 below 50%, highlighting the substantial challenges of scientific software engineering. The study identifies four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases.
A paired ablation that removes explicit scientific guidance while preserving repository and executable context shows that scientific knowledge is not uniformly beneficial. Well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success.
The benchmark uses a separate private test suite to evaluate scientific contracts, preventing agents from gaming public tests. For example, Qwen3.8-27B passed all 119 public reproductions but achieved full private Pass@1 on only 35 of 119 tasks.