Researchers from Hugging Face have published a paper titled 'Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation,' shedding light on a subtle but critical failure mode that arises when distilling diffusion models that use classifier-free guidance (CFG).
On-policy distillation (OPD) adapts a teacher diffusion model into a student by querying the teacher along trajectories generated by the student. A common approach is to match the teacher's and student's guided predictions. However, the authors show that this objective is under-identified at the branch level: errors from the positive and negative CFG branches can cancel each other out, making the guided output appear correct even when individual branches are not.
Through analysis, the team identified two contrasting cases. When teacher and student share the same negative conditioning, naive matching works well because both branch errors decrease jointly. But when the teacher's native CFG schema retains privileged information in its negative branch that is unavailable to the student, the branches can start moving in opposite directions during training. This failure mode is termed Negative Branch Asymmetry (NBA).
To address NBA, the researchers propose Positive–Direction Matching (PDM), a branch-aware objective that separately constrains the positive prediction and the CFG conditional direction. This removes cross-branch error compensation and makes the distilled model substantially more robust to changes in inference guidance scale.
The study validates PDM on image-domain diagnostics and dense-to-sparse video control tasks (including pose, depth, and scribble conditions). Results show that while naive guided matching is highly sensitive to inference guidance scales, branch-aware supervision enables more robust and effective knowledge transfer.