On-policy distillation (OPD) is a widely used technique for improving reasoning models: a student model learns from the token-level discrepancy between its own on-policy outputs and those of a stronger teacher. A new research paper argues that this discrepancy is not a clean measure of the capability gap between the two models.
According to the authors, the observed teacher-student discrepancy also contains deviations that originate from the teacher itself. Standard OPD learns this mixed signal indiscriminately, so the student can end up imitating the teacher's own errors rather than only its superior behavior. The problem is described as more severe in privileged OPD, where privileged information causes larger shifts in the teacher's likelihood and pushes the student to absorb even more of the teacher's self-deviation.
To address this, the paper introduces Calibrated On-Policy Distillation (Cal-OPD). The method estimates the teacher's self-deviation region using both positive and negative privileged interventions, then calibrates the original teacher-student discrepancy by keeping only the component that falls outside that region.
Experiments on mathematical reasoning benchmarks show that Cal-OPD consistently outperforms standard OPD and its variants across model scales — even though it retains only about 52–65% of the original teacher-student discrepancy as the optimization signal.
The authors frame the work as a study of how part of the teacher signal can reflect teacher self-deviation rather than useful supervision, and report that intervention-based calibration improves on-policy distillation across multiple teacher-student pairs.