Reinforcement learning with verifiable rewards (RLVR) has become a key technique for training large language models, but a new paper from Hugging Face identifies a critical flaw in standard approaches. Methods like Group Relative Policy Optimization (GRPO) typically apply a fixed importance-sampling (IS) ratio clipping boundary across all rollouts, which can inadvertently suppress the most valuable learning signals.
The researchers observed that rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success—where few correct solutions exist within a group—tend to have larger IS ratios and carry stronger gradient signals for exploration and solving new problems, yet they are disproportionately suppressed by uniform clipping.
To address this, the team proposes Group Adaptive Clipping Policy Optimization (GAPO), a simple plug-in modification that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, suggesting that rollouts with larger learning signals should receive proportionally greater update headroom. Importantly, GAPO requires no reward shaping or objective modification, preserving the standard PPO/GSPO surrogate while adapting only the clipping threshold.
Across Qwen and Llama base models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on mathematical reasoning and coding benchmarks where base model pass rates are relatively low. The paper is available on Hugging Face and has attracted related recommendations from the Semantic Scholar API.