Hugging Face has released a technical report introducing KVAE, a family of tokenizers tailored for multimodal generative models. Tokenizers play a crucial role in latent diffusion modeling (LDM) by compressing input signals into a latent space, directly influencing learning speed, sample quality, and downstream applications.
The KVAE family includes three specialized tokenizers: KVAE-Audio, a continuous full-band 48 kHz audio tokenizer with a 50 Hz latent rate and 64 channels; KVAE-3D, two causal video tokenizers offering 4x16x16 and 4x8x8 compression; and KVAE-2D, an image tokenizer that compresses input by a factor of 8 with 32 channels. All are designed for text-conditioned generation.
According to the report, KVAE tokenizers match or surpass frontier open-source tokenizers such as those from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio, and MMAudio. The evaluation covers reconstruction metrics (PSNR, LPIPS, PESQ) and generation metrics (Frechet Distance, CLIP score, CLAP score), as well as subjective side-by-side comparisons.
The authors share training details, model selection methods, and ablation studies on design choices to aid further development. Inference code and weights are publicly available on GitHub and Hugging Face Hub.