Researchers from Zhejiang University and Hugging Face have introduced Relay-OPD, a novel on-policy distillation method that addresses a critical weakness in training small language models from larger ones: prefix failure. When a student model commits to an incorrect reasoning direction early in a generated sequence, all subsequent tokens build on that mistake, leading to unreliable supervision and wasted computation.
The key insight is a measurable asymmetry between teacher and student on failed prefixes: the teacher tends to produce redirective tokens (e.g., "But", "Wait") while the student continues along the original erroneous path. This divergence can be detected online without any external verifier, reward model, or labels, serving as a natural handoff trigger.
Relay-OPD constructs relay trajectories by letting the teacher briefly take over at these trigger points to produce a "teacher leg," after which the student resumes and is optimized on the resulting trajectory. A limited relay budget ensures intervention concentrates on critical early positions while minimally departing from the student's policy. The entire process runs within a single speculative-decoding engine where the student drafts and the teacher verifies.
Experiments used a Qwen3-4B-Instruct-2507 teacher with Qwen3-0.6B and Qwen3-1.7B non-thinking students across eight mathematical reasoning benchmarks. Relay-OPD achieved the best or second-best results on every benchmark, outperforming standard OPD by an average of 5.73% and the strongest baseline FastOPD by 1.49% for the 1.7B student, with consistent gains at 0.6B. Training trajectory length was reduced by over 50%.
Code and project details are available on GitHub and the project website.