A new research paper posted on Hugging Face examines a long-standing assumption in model training: that on-policy learning — where a model learns from its own generated outputs — is inherently superior to off-policy learning. The authors argue that prior comparisons between supervised fine-tuning and reinforcement learning changed too many variables at once, making it impossible to isolate the true effect of rollout policy.
To address this, the team designed a controlled strong-to-weak distillation setup. They independently varied three factors: the rollout policy, the direction of token-level KL divergence, and the learning rate. Experiments spanned the Llama3 and Qwen2.5 model families and covered reasoning tasks in scientific, medical, and arithmetic domains.
The results paint a more nuanced picture than expected. Rollout policy did not necessarily play a central role. Instead, the direction of the token-level KL objective more clearly shaped task performance and output coverage, while the learning rate was the dominant factor governing catastrophic forgetting and update sparsity.
Analysis of KL gradients and experiments along a continuous student-teacher rollout-policy spectrum helped explain the pattern. Forward KL proved remarkably robust to changes in rollout policy, maintaining stable and strong performance. Reverse KL, by contrast, was substantially more sensitive and favoured student-generated rollouts.
On-policy data did improve generalisation to harder variants of the Countdown arithmetic task under both KL directions — but this advantage did not reliably persist after subsequent reinforcement learning with verifiable rewards (RLVR). The authors report that their broader conclusions held even after removing gradient clipping, using sampled KL estimators, and training on tasks requiring longer reasoning chains.
Overall, the study challenges the view that on-policy rollouts are inherently preferable, concluding that their value depends critically on the objective, the evaluation setting, and optimisation hyperparameters.