Few-step autoregressive (AR) video diffusion promises low-latency streaming generation, but the dominant post-training route — Distribution Matching Distillation (DMD) — depends on a large pretrained teacher plus an online critic that estimates distributional gaps through diffusion scores. A new paper, ViRDM, asks whether that resource-heavy teacher–critic stack can be removed entirely by post-training only the generator against a precomputed target distribution.
Drawing on representation distribution matching (RDM) as used for one-step image generation, the authors systematically study its transfer to few-step causal video generation and identify three barriers: a memory-intractable gradient path, a distinct video optimization regime, and representation distributions that underconstrain temporal dynamics.
ViRDM addresses these in sequence. It couples RDM with stochastically truncated clean-exit supervision, a lightweight VAE decoder, and staged vector–Jacobian products to make representation distribution matching memory-feasible for multi-step causal video rollouts. The authors also establish effective generated-population and initialization regimes for video RDM, and add lightweight dynamics regularization to compensate for underconstrained temporal motion.
The result converts three-network distillation into generator-only post-training, reducing GPU memory use and training time while improving video quality. With only 20 generator updates, the recipe reaches 84.87 on the official VBench evaluation, outperforming the previous best few-step causal baseline by 0.36, while requiring 16 A100 GPU-hours — roughly two hours of post-training on 8×A100.
The team notes the recipe also supports training on a single 80GB A100, a setup that is challenging for common DMD pipelines. Exploratory results further suggest the same recipe can extend to a lower causal sampling budget and to one-, two-, and four-step bidirectional generation.
Releases include training and inference code, model weights and precomputed reference artifacts, reproduction configurations and evaluation scripts, and a single-80GB-GPU training launcher. The authors credit prior work including FD Loss, RDM, Self Forcing, and Causal Forcing as foundations.