As AI coding agents tackle increasingly complex software engineering tasks, existing benchmarks are saturating, and their evaluation quality is under scrutiny. A recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, and frontier models can reproduce gold patches from training data. This highlights the need for more robust benchmarks.
To address this, Hugging Face researchers introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark. It consists of 170 instances drawn from real commits across seven programming languages: Python, Java, TypeScript, Go, C, C++, and Rust. Each instance undergoes rigorous curation to fix quality issues, with rewritten issue descriptions and manually reviewed test suites to remove overly narrow or broad tests.
The benchmark focuses on large-scale refactoring tasks, averaging 11.4 modified files and 261.6 lines of code per instance—substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show the best model achieves only a 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents.
The benchmark is publicly available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.