On-policy distillation (OPD) has been proposed as an alternative to reinforcement learning with verifiable rewards (RLVR), offering dense token-level supervision. However, a new paper from Hugging Face questions the source of OPD's effectiveness, suggesting that its gains may not come from teacher guidance at all.
The researchers quantitatively analyzed teacher supervision during OPD training and found substantial noise, which increases with teacher scale. Surprisingly, the student policy is insensitive to this noise, converging to comparable performance whether noisy supervision is retained or removed. This raises the question: does OPD actually distill?
By analyzing what drives OPD's gains, the authors found that learning concentrates on low log-probability tokens. Using a single fixed negative advantage matches the performance of teacher-provided ones, suggesting that OPD works largely by suppressing low-probability tokens, which requires no teacher.
Based on these findings, the paper introduces On-Policy Self-Adaptation (OPSA), a supervision-free method that uses entropy-adaptive negative advantages. OPSA assigns stronger learning signals to high-entropy positions, suppressing tail tokens and evenly redistributing probability mass among head tokens.
Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24 (a 263% relative gain) and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments across model families and tasks demonstrate its effectiveness and generalizability.