Autoregressive (AR) video diffusion models have emerged as a promising route to real-time video generation. A common approach distills pretrained bidirectional video diffusion models into causal AR students using Distribution Matching Distillation (DMD). However, according to a new paper posted on Hugging Face, the resulting videos often show over-saturation and over-smoothing, limiting both visual quality and realism.
The authors trace the problem to the mode-seeking behavior of the reverse KL objective used in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution.
To address this, they introduce Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student's self-rollout. The core idea is to inject cleaner signals into noisy rollout inputs through random masks applied along spatial and temporal axes during self-rollout.
These perturbations, the paper argues, push student rollouts to explore more regions of the teacher distribution, letting DMD supply learning signals beyond the modes the student already covers. In addition, the cleaner tokens act as denoising guidance for noisier tokens, improving intermediate rollout predictions and reducing error accumulation.
The authors report that extensive experiments improve multiple AR video diffusion distillation methods with higher visual quality and efficiency, without incorporating real video data or adding post-training stages.