Researchers have proposed Domain-Normalized Multi-Teacher On-Policy Distillation (DN-MOPD), a method for merging several specialist language models into a single student model. The work, published as a Hugging Face paper, addresses a limitation in multi-teacher on-policy distillation (MOPD), where specialists in domains such as mathematics, coding, and instruction following each provide token-level feedback on prompts routed to their domain.
The authors found that in Qwen3.5 models at three sizes, MOPD's student did not outperform a student taught by the best single specialist and captured little of the mathematics specialist's advantage. The cause, they report, is unbalanced feedback: instruction-following feedback was several times more spread out than mathematics feedback, causing it to dominate the student's updates.
DN-MOPD keeps the existing routing but rescales each domain's feedback according to its measured spread. On six public benchmarks, the method improved the average score over MOPD at every model size, across three random seeds and under two answer-length limits, while recovering most of the lost mathematics gain.
Control experiments with fixed domain weights indicated that the improvement stems mainly from reducing instruction-following feedback rather than increasing mathematics feedback alone. Fixed weights close to those measured by DN-MOPD performed comparably, suggesting that the key is balancing feedback strength, not only selecting which specialist teaches.
The paper concludes that combining specialists requires deciding both which one teaches and how strongly its feedback counts. A project page and code repository are available.