On-policy distillation (OPD) has emerged as an effective paradigm for transferring capabilities across large language models (LLMs), aligning a student model with the teacher's token-level distribution on the student's own rollouts. However, prevailing approaches assume a teacher at least as capable as the student, which fails at the frontier where no larger teacher exists, or they consolidate multiple domain experts trained from a shared base, requiring costly training at the student's scale.
To address this, researchers from Hugging Face introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective framework that improves a strong student by distilling from multiple weak models. The method constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is added to the student's own base model, yielding a proxy teacher that couples this direction while staying distributionally adjacent to the student.
The student then distills this proxy teacher by minimizing the per-token reverse KL divergence on its own rollouts. The contrast pair can be instantiated in three ways: (i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills; (ii) a larger against a smaller base model, isolating the capability from scale; and (iii) a small base model with correct versus wrong hints, isolating the instance-level direction toward the solution.
Across four math and three code benchmarks, W2S-OPD outperforms standard OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker. Analysis reveals that different contrasts yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. The code is available on GitHub.