Reinforcement learning (RL) has proven effective for code correctness, where models generate programs and are rewarded for passing hidden tests. Extending this to code optimization—rewarding faster execution—has been challenging due to measurement noise, reward sparsity, and instability in GRPO, often leading to marginal speed gains and more failures.
In a new paper, Hugging Face researchers propose a three-stage approach to make execution time learnable. First, they build DMC-Optim, a benchmark with large optimization tests and a calibrated sandbox for reliable timing. Second, they design a reward function that combines correctness and speed, using an offline simulator to predict the most promising configurations. Third, they adapt GRPO and evaluation to handle the sparser, noisier timed-execution setting.
Results on DMC-Optim show significant improvements: strict top-50% pass@1 rises from 18.0% to 31.3% on Qwen 2.5 7B and from 30.7% to 50.4% on CWM 32B. At stricter percentiles like top-30%, CWM 32B achieves a 125% relative improvement while preserving pure-correctness scores. When the timing sandbox is degraded, robust optimization RL outperforms standard RLVR by 100% to 200% depending on the evaluation criterion.
On the LCB benchmark, CWM 32B wins up to 83% of median-sample speed comparisons against standard RLVR. Relative to the fastest correct human submissions per problem, it achieves about half the human rate of complexity-class improvements (14% vs. 28%).