A newly posted paper on Hugging Face examines why on-policy post-training paradigms tend to generalize better than supervised fine-tuning (SFT), and proposes a method that transfers that advantage to SFT by controlling the direction of parameter updates.
According to the authors, prior work has treated the update behaviors observed during on-policy training largely as byproducts rather than as potential optimization principles. Their analysis finds a key difference: SFT updates parameters along consistent directions, whereas on-policy training continuously adjusts the update direction during training. This leads them to focus on each parameter's cumulative update direction as a promising behavior to exploit.
To test this idea, the paper introduces On-Policy direction-constrained Supervised Fine-Tuning (OPSFT), which constrains SFT updates to the direction identified by on-policy paradigms. The authors report that OPSFT performs strongly, suggesting the generalization advantage of on-policy training can be transferred to SFT through the parameter update direction — once such a direction is identified, even SFT can generalize when its updates are constrained to it.
The paper highlights two practical benefits. For efficiency, update directions that support strong generalization can be identified using only a few on-policy training steps, after which OPSFT delivers high training efficiency. For leveraging high-quality trajectories, OPSFT can use such trajectories to continue improving a post-trained model along its update direction without disrupting the abilities learned during on-policy training.
The work is framed as an answer to the question of why on-policy training generalizes better than SFT, pointing to update direction as a crucial factor. The paper page also lists related recent work surfaced by automated recommendation tools, including studies on on-policy distillation, off-policy intervention, and rollout reinforcement learning.