Sparse on-policy distillation (OPD) gives teacher supervision to only a small subset of tokens in trajectories generated by the student model. The idea is efficient, but the paper argues it has a hidden flaw: even when teacher guidance is genuinely useful, the update can become noisy because the gradient is estimated from a single sampled next token.
To address this, the authors study the estimation problem at a fixed prefix through the lens of information geometry. They introduce an information-efficiency ratio (IER), derived from a signal-to-noise decomposition, which characterizes the relative gradient estimation error under an optimal scalar baseline.
The team then develops a candidate-set approximation that allows tokens to be selected based on IER, either alone or combined with existing usefulness scores, while keeping the sampled reverse-KL training objective intact.
Experiments on mathematical and medical reasoning tasks show that adding IER improves existing token selectors across multiple settings. Notably, sparse configurations with token budgets of just 0.1%–1% matched or exceeded full OPD without token selection.
The authors conclude that sparse supervision should weigh both how useful a token is and how reliably its gradient can be estimated. Code is available on GitHub.