Researchers have introduced Latent-MOPD, a method they describe as the first representation-level multi-teacher on-policy distillation (OPD) approach for large language models. The work appears as a paper on Hugging Face.
Conventional multi-teacher OPD transfers knowledge from specialist models only through their output distributions. Latent-MOPD instead integrates specialists through both their predictions and the hidden states used to compute those predictions, without requiring any additional teacher training.
To coordinate supervision from multiple specialists, the method selects late-layer targets based on the teacher-student relationship, bridges unequal hidden widths with a shared projection, and groups updates by domain. Each teacher's supervision gradually shifts from hidden states toward token predictions, with both channels routed to the same specialist.
In the main same-family setting, Latent-MOPD reportedly outperforms token-only, representation-only, and uniform-averaging baselines on all nine benchmarks spanning math, code, and logic. With the same parameter count as each teacher, the student also surpasses the per-benchmark best teacher on a majority of these benchmarks.
With larger, separately developed cross-family teachers, Latent-MOPD outperforms both single-channel baselines on all benchmarks. A same-family all-layer representation-only control remained stable with domain-pure updates but collapsed when teacher domains were interleaved within an update.
The authors conclude that a single student can integrate capabilities from several specialists through both their output distributions and internal representations.