A new research paper introduces Influence-Directed Adaptive On-Policy Distillation (IDA-OPD), a method designed to overcome the diversity bottleneck in sampled-token on-policy distillation (OPD) for reasoning models. The work, posted on Hugging Face, addresses a known failure mode where student models improve in pass@1 but fail to match the teacher's diversity, as measured by pass@k.
The authors first propose First-Order Local Entropy Influence, a signed first-order proxy that decomposes each update's entropy effect into two components: the teacher-student log-probability gap and the student's local probability structure. This analysis empirically links entropy contraction to positions with negative influence, providing a theoretical foundation for the new method.
Instead of relying on costly full-vocabulary Forward-KL objectives, IDA-OPD selectively preserves entropy-expanding updates while replacing entropy-contracting ones with divergence-adaptive advantage shrinkage. This approach only requires the teacher's sampled-token log-probability, making it computationally efficient.
Experiments on reasoning-oriented distillation tasks show that IDA-OPD consistently improves pass@k, successfully inheriting the teacher's diversity. It matches the performance of the strongest teacher-informed methods at strictly lower cost and broadly maintains vanilla OPD's pass@1, all without needing full-vocabulary teacher information.