Reinforcement learning with verifiable rewards (RLVR) is known to improve single-sample accuracy (pass@1) but often causes the policy's solution space to contract, reducing the benefits of test-time scaling. A new paper investigates where this loss of breadth occurs within a reasoning trajectory: whether the policy fails to access a valid solution family or fails to execute computation once initiated.
To disentangle access from execution, the researchers analyzed the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator. They trained PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct, finding that solution coverage fell by up to 67%, and even halved on problems solved across all checkpoints.
The contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x–16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restored completion rates in low-access families by over an order of magnitude (from 0.018 to 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated.
Guided by this localization, the authors found that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increased solution coverage by 37% with no loss in pass@1. They also showed that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not inevitable: an SFT baseline preserves more than double the coverage, and staged SFT–DPO–RLVR pipelines retain early-step entropy.
In summary, reasoning breadth is lost at the door, not inside the room. The code is available on GitHub.