On-policy distillation (OPD) has become a common approach for post-training large language models, supervising a student model on its own generated trajectories to reduce the mismatch between training and testing. However, the paper argues that existing OPD objectives remain largely token-local and outcome-agnostic: they optimize teacher-student agreement at each prefix even though reasoning quality is determined at the trajectory level.
Reinforcement learning with verifiable rewards (RLVR), especially Group Relative Policy Optimization (GRPO), offers complementary outcome-level supervision but suffers from sparse rewards and coarse credit assignment. The authors show that OPD and RLVR have complementary blind spots: teacher signals give dense local guidance but are weakly aligned with rollout correctness, while group-relative rewards capture task success but provide coarse token-level credit and vanish on all-failure groups.
To address this, the paper introduces DiffGate, an outcome-gated objective that combines GRPO with selective, bounded teacher guidance. Teacher supervision is applied only to failed trajectories, scaled by group difficulty, and smoothly bounded to prevent extreme teacher-student discrepancies from dominating optimization. The verifier thus determines which trajectories receive teacher guidance, while the teacher supplies dense token-level update directions within those trajectories.
Experiments across Qwen3-0.6B and Qwen3-1.7B students show that DiffGate improves code avg@8 over matched GRPO by +1.7 and +1.8 points, and pass@8 by +1.6 and +5.7 points, respectively. On mathematics, avg@8 remains within 0.5 points of GRPO while pass@8 improves by +1.1 and +3.9 points. Overall, DiffGate improves pass@8 across all four model-domain settings, indicating better solution coverage under the evaluation protocol.