Latent diffusion models have become a leading approach for high-fidelity image and video synthesis, operating in compact latent spaces using variational autoencoders (VAEs) to improve computational efficiency without sacrificing visual quality. However, conventional VAEs are not well-suited for video data because they use fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content.
To address this, researchers from Kakao and other institutions introduce KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector jointly learned with latent tokens. The selector evaluates each token's content-richness as a keep-or-drop probability, discarding uninformative tokens and enabling data-dependent compression.
Applying adaptive tokenization to diffusion models can cause spatial misalignment, as dropping tokens may disturb the original spatio-temporal structure. To mitigate this, the authors propose two position-prediction strategies: cascaded and joint generation, ensuring spatial consistency.
Empirical results show that KATok achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis indicates that the improvement primarily comes from reducing spatio-temporal redundancy and removing uninformative tokens, supported by both quantitative and qualitative results.
For more details, visit the project page or watch the YouTube video.