On-policy distillation (OPD) has proven effective in large language models (LLMs) and has recently been adapted to flow matching models, but it typically relies on a pre-trained, task-specific teacher to provide dense supervision. This approach incurs high computational costs for each new objective and suffers from compounding errors due to teacher-student distribution mismatch.
In a new paper, Hugging Face researchers propose Self-OPD, a teacher-free OPD framework that turns the student's own self-exploration into step-wise supervision. At each timestep, the deterministic next-state prediction is branched into K stochastic SDE candidates, which are rolled out with the ODE sampler. Rewards from these branches are compared against a deterministic self-reference baseline to compute normalized advantages.
The velocity field is optimized using an all-branch pull-push objective: high-advantage branches attract the student, while low-advantage branches repel it, with direction-aware attenuation and SDE-variance normalization. For multi-objective alignment, Self-OPD fuses normalized scores at the reward level, avoiding direct gradient conflict.
Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without requiring task-specific teachers, offering a more efficient and robust alternative for flow matching distillation.