Researchers at Hugging Face have introduced V-RAE (Video Representation Autoencoder), a novel approach to video latent spaces for generative modeling. The work addresses a key limitation of conventional video autoencoders: their latent spaces are optimized for pixel-level reconstruction, not for the semantic organization that generative models need.
V-RAE builds compact generative latents on top of frozen vision foundation model representations, such as DINOv3, SigLIP2, EUPE, and V-JEPA 2.1. A lightweight temporal pooling module compresses frame-level representations while preserving semantic and motion information, and a video decoder reconstructs continuous motion from the compressed features.
In evaluations, V-RAE achieved a reconstruction FVD of 2.13 on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, the best variant achieved gFVD scores of 117.86 on UCF101 and 19.16 on K600, while converging up to 6x faster.
The authors also introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality than reconstruction FVD, highlighting that reconstruction quality alone is insufficient to characterize generative utility. Beyond generation, V-RAE improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched settings.