Researchers from Hugging Face have published a paper exploring a fundamental question in AI: can stronger models learn from weaker supervisors and ultimately surpass them? This challenge, known as weak-to-strong generalization, is crucial for successive model generations and multi-domain consolidation, where retraining frontier-scale models from scratch is often prohibitively expensive.
Traditional distillation methods treat the weak teacher as an optimization target, which can inadvertently cap the student's performance at the teacher's level. To address this, the team introduces On-Policy Reverse Distillation (OPRD). This technique evaluates the teacher's policy shift relative to its reference policy on student rollouts and amplifies the component of the student's verifier-driven policy gradient along that direction.
By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. The authors demonstrate that in both successive model transfer and multi-teacher distillation scenarios, OPRD achieves higher performance with fewer student updates compared to existing reinforcement learning and distillation approaches.
Response-style analysis reveals that OPRD students remain closer to models trained with verifier-based RL alone than to their weak teachers, indicating that teacher guidance accelerates rather than redirects the student's own optimization. The method also proves effective in conventional strong-to-weak distillation, showing its versatility regardless of capacity ordering.
The paper outlines three key contributions: studying weak-to-strong generalization in practical scenarios, introducing the OPRD algorithm, and providing empirical evaluation showing OPRD reaches final performance earlier and outperforms baselines.