On-policy distillation, where a teacher model corrects samples generated by the student, traditionally requires both models to share the same latent space, architecture, and timestep schedule. This assumption breaks when distilling between different model families, such as distilling a large model into a smaller one from a different lineage. The standard recipes fail: teacher latents cannot serve as targets in a foreign coordinate system, per-pixel losses against a teacher that stochastically re-draws local detail lead to blur or divergence, and timestep indices lose meaning across mismatched schedules.
To address this, researchers present Any-OPD, the first framework for on-policy distillation between arbitrary pairs of latent flow-matching generators. Any-OPD treats the teacher purely as a black-box sampler and connects the two models at exactly one point: a frozen, model-agnostic vision representation in which their independently decoded outputs are compared. This sidesteps every assumption about latents, features, or architecture.
Trajectory correspondence is recovered by matching continuous noise levels instead of step indices. A brief anchoring phase, where teacher samples are re-encoded through the student's own VAE, ensures the on-policy gradient measures sample quality rather than domain mismatch.
In experiments, distilling the 12B FLUX.1-dev into the 2.5B SD3.5-Medium, Any-OPD lifts the student's PickScore from 0.846 to 0.884 and HPSv3 from 9.12 to 10.97, rivaling the teacher at a fifth of its size. Direct latent regression fails to train at all in this setting.