Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) each have complementary strengths: RLVR broadcasts a single response-level reward, while OPD provides dense token-level advantages from a stronger teacher. However, naively fusing the two with a fixed coefficient leads to entropy collapse, as the team from Hugging Face discovered.
The problem stems from two miscalibrations: a magnitude mismatch, where OPD advantages can spike far beyond the bounded RLVR advantage, and a temporal mismatch, where sustained full-strength distillation limits exploration needed to surpass the teacher.
To address this, the team introduces Stable Advantage Fusion (SAF), a lightweight four-stage pipeline applied only to the OPD advantage: sparsify-then-compress for magnitude control, and warm-up-then-anneal for temporal control. Each stage is independently switchable and adds negligible overhead.
Evaluating with GRPO across seven math and code benchmarks using Qwen3-1.7B/4B/8B, SAF avoids entropy collapse and consistently outperforms fixed-coefficient fusion, improving aggregate scores by 0.51–2.70% across all six model-domain settings while achieving more stable training.