On-policy distillation (OPD) is a training technique in which a student model learns from its own generated rollouts while receiving token-level supervision from a teacher model. The approach assumes the teacher can reach a correct answer from any prefix the student produces — a property the authors call recoverability.
The paper shows that this assumption breaks down on incorrect rollouts. Although the teacher recognizes that the student has made an error, it tends to follow that error when continuing the flawed solution, making the supervision unreliable.
To address this, the researchers propose CriticOPD, which uses the teacher's error recognition to restore recoverability. When a rollout is incorrect, the teacher critiques it using the reference answer, and the student then rewrites its solution following the critique. The teacher scores the rewritten output with the critique included as privileged information, while the student trains on the revision without seeing the critique.
Experiments on two Qwen3 student models and six mathematical reasoning benchmarks show that CriticOPD improves over standard OPD. Average accuracy rises by 4.57 percentage points on Qwen3-1.7B and 4.06 points on Qwen3-0.6B, with the largest gains observed on competition-level problems.