On-policy self-distillation (OPSD) has shown promise for improving reasoning language models, but its practical application is often brittle and requires significant engineering effort. A new paper identifies a structural cause: vanilla OPSD is exactly the β=1 case of a broader policy-optimization family, where β weights the KL penalty that anchors the student to a reference policy.
By making β a controllable regularization parameter, the authors derive β-OPSD, whose optimal policy is a geometric interpolation between the reference policy and a privileged teacher. Rather than directly optimizing the costly and high-variance reinforcement learning objective, they convert its closed-form solution into an efficient token-level distillation target, mixing logits from the reference and teacher according to β.
This approach approximates expensive policy optimization with inexpensive distillation, while return-to-go credit assignment aligns token updates with sequence-level outcomes. Experiments on mathematical reasoning benchmarks with Qwen3 models (1.7B to 8B) show that β-OPSD consistently outperforms vanilla OPSD, improving both optimization stability and downstream reasoning performance.
The authors report a +5.74-point average gain at the 1.7B scale, highlighting the practical benefits of their method. The work provides a principled bridge between self-distillation and policy optimization, retaining the efficiency that makes OPSD practical.