SWE-Bench Pro has become a widely used standard for testing software engineering agents on difficult repository-level tasks. But according to a new paper, its evaluation is undermined by two sources of unreliability: reward hacking, made possible by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests.
The authors argue these flaws can inflate benchmark scores and obscure an agent's true coding ability. To address them, they introduce SWE-Bench Pro Verified, a verified version of the benchmark.
The approach combines anti-hacking safeguards that close major leakage channels without disrupting normal agent functionality, alongside task refinement that makes minimal corrections to inconsistencies within flawed instances.
Evaluations on SWE-Bench Pro Verified show that some frontier models perform substantially worse than previously reported, suggesting that existing SWE-Bench Pro results may overestimate real software engineering capability. The authors position the verified set as a more trustworthy benchmark for assessing software engineering agents.