On-policy distillation (OPD) trains a student model on its own generated outputs using feedback from a teacher. When teacher and student use different tokenizers, comparing their predictions requires alignment at both the sequence and vocabulary levels. A new paper on Hugging Face asks whether expanding that alignment coverage actually improves learning.
Across three heterogeneous teacher–student pairs evaluated on mathematical reasoning and code generation, the authors report that strict 1:1 alignment groups already cover most student-generated tokens despite substantial vocabulary mismatch. On responses sampled from the students before distillation, the shared vocabulary retained nearly all teacher and student probability mass at strictly aligned positions on average.
The study finds that restricting reverse KL to a student-selected top-16 subset of the shared vocabulary at each strict position achieves accuracy comparable to full shared-vocabulary OPD, and outperforms the evaluated cross-tokenizer baselines. In contrast, adding mean squared error supervision on span log-probabilities in mismatch groups provides complete supervision coverage but reduces accuracy.
The authors offer a diagnostic explanation: at checkpoints trained with only the strict loss, span gradients show weak or negative directional agreement with the strict gradients and grow in magnitude relative to them. These diagnostics may help explain the accuracy drop from adding span supervision.
The paper concludes by motivating a shift from maximizing alignment coverage to prioritizing supervision reliability, arguing that compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals.