Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for post-training large language models (LLMs). However, a widely used method called Group Relative Policy Optimization (GRPO) suffers from sparse reward signals and loses gradients entirely when all responses in a group receive identical rewards. This can stall learning.
To address this, researchers from Hugging Face propose a new method called RSTG (Recovering Learning Signals via Adaptive Teacher Guidance). RSTG builds on on-policy distillation (OPD), which provides dense, token-level supervision from a teacher model. But naive combinations of GRPO and OPD often degrade performance due to three issues: not all samples benefit from distillation, overly fast fitting to the teacher reduces exploration, and OPD's advantages are asymmetric, suppressing most tokens.
RSTG applies distillation selectively. At the sample level, it restricts OPD to negative zero-variance prompts, weighting each sample by the teacher's confidence. At the token level, it targets only tokens with high student entropy or large teacher-student divergence. Additionally, it augments training with SFT on correct teacher-generated trajectories, injecting positive gradients where RL provides none.
Experiments show that RSTG substantially outperforms naive GRPO+OPD, achieving a +4.02% improvement on math tasks and +3.05% on code tasks. The paper is available on Hugging Face and has been recommended alongside related works in the field.