Reinforcement learning can unlock substantial reasoning capabilities in large language models, but how much of that capability carries over across model scales — and how quickly — has remained an open question. A new paper posted on Hugging Face, titled Scaling Properties of Same-Family On-Policy Distillation, examines this problem through the lens of on-policy distillation (OPD).
The authors study three teacher–student configurations: weak-to-strong, same-base, and strong-to-weak. Across these setups, they report that early OPD training dynamics consistently show a regular useful-transfer regime, in which held-out accuracy — referred to as the gold score, G — rises approximately linearly with d = KL(π_θ ‖ π_ref), the square root of the token-level reverse KL divergence from the student's initialization.
One notable finding is that in every observed weak-to-strong pair, the student's peak gold score exceeded that of its own teacher. According to the paper, this means a compact RL expert can transfer capability to a much larger student through OPD.
To predict OPD outcomes, the researchers fit power laws describing how the peak gold score and the slope of the useful-transfer regime scale with student and teacher parameter counts and with teacher gold score. These laws indicate that peak gold score improves with teacher scale only up to roughly the student's scale, and that at a matched gold score, smaller teachers transfer better — suggesting a teacher's score alone does not define its supervision value.
The paper also explores the scaling effects of two OPD variants: bootstrapping weak-to-strong OPD and the degree of on-policy supervision. Alongside the paper, the authors released an interactive project page with all 25 gold-score trajectories, a training replay, the full 5×5 peak grid, and a calculator that predicts peak performance from student size, teacher size, and teacher score. Training checkpoints covering teachers, students, and baselines — 311 models spanning Qwen2.5 0.5B to 14B — are also available on Hugging Face.