On-Policy Distillation (OPD) is gaining traction as an alternative to reinforcement learning for post-training large language models, but its effectiveness in multilingual contexts has been largely unexplored. A new paper from Hugging Face investigates OPD and its advanced variant, On-Policy Delta Distillation (OPD²), specifically for mathematical reasoning in English, Korean, and Japanese.
OPD² improves upon OPD by using the probability gap between a post-trained teacher model and its base model as the learning signal, rather than relying solely on the teacher's outputs. In experiments with the Qwen3 model family, OPD² consistently outperformed the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrowed the performance gap between English and Korean.
The study also found that English-only OPD can boost performance for Korean and Japanese, but it often shifts the model's responses toward English, underscoring the importance of including multilingual data to preserve target-language responses. This highlights a key trade-off for practitioners aiming to improve reasoning in non-English languages.
The findings suggest that OPD² offers a promising path for efficient, multilingual post-training, potentially reducing the need for extensive reinforcement learning pipelines while maintaining or improving reasoning quality across languages.