On-policy distillation (OPD) is a technique where a teacher model provides dense supervision on trajectories generated by a student model. However, standard reverse-KL training can assign insufficient probability to other plausible continuations, limiting the student's ability to explore diverse solutions.
To address this, researchers from Hugging Face introduce SPOT (Sparse Probing and Outcome-calibrated Targets OPD). SPOT tackles two coupled decisions: where to probe and what to distill, through an acquisition–exploration–exploitation procedure. During acquisition, a position-level score combines normalized teacher entropy, the probability mass captured by a small top-k candidate set, and student–teacher mismatch to allocate a limited probing budget.
During exploration, SPOT evaluates teacher-proposed candidates through verifier-scored student continuations. During exploitation, these outcomes produce a closed-form, KL-regularized target that favors candidates with better downstream outcomes while remaining anchored to the teacher distribution.
Extensive experiments across multiple student models and reasoning benchmarks demonstrate SPOT's effectiveness. Across three student scales, SPOT improves macro Avg@8/Pass@8 over OPD by 0.47–1.48/4.55–5.28 points and over EOPD by 0.29–0.68/2.49–3.19 points, balancing solution quality and coverage.