On-policy distillation (OPD) is a technique that transfers capabilities from a teacher model to a student by supervising trajectories sampled by the student with dense, token-level teacher signals. Recent selective OPD methods aim to improve this process by prioritizing signals that are confident, informative, or learnable. However, these methods overlook a fundamental failure mode: token-level judgments from language models can be driven by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates rather than task-specific evidence.
To address this, researchers introduce SA-OPD (Spurious-Signal-Aware On-Policy Distillation), a framework that identifies and filters misleading token-level supervision based on two criteria: input-groundedness and optimization impact. SA-OPD uses a lightweight proxy to estimate whether a token-level distillation signal truly depends on the input. It then filters only tokens that exhibit both low input-groundedness and extreme distillation divergence, removing high-impact spurious updates while preserving useful supervision.
The method compares token-level teacher–student divergence under the original input and a residual no-prompt context, providing a simple proxy for input-groundedness without requiring external verification labels or auxiliary judges. This targeted filtering aims to avoid large gradients that contribute little to task improvement.
Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods on benchmarks for mathematical reasoning, visual understanding, and visual reasoning. The results establish input-groundedness as a key dimension for OPD supervision selection, offering a practical strategy for more reliable distillation.