Recent advances in generative AI have pushed beyond separate audio or video synthesis toward joint generation of synchronized audiovisual content. However, achieving fine-grained cross-modal correspondence remains difficult due to fundamental structural differences between audio and video modalities. Most existing approaches rely on independently trained audio and video VAEs, forcing downstream models to learn cross-modal synchronization from scratch.
In a new paper, Hugging Face researchers present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. The model incorporates two complementary objectives during VAE training: segment-level audio-video contrastive learning and modality-specific semantic distillation.
The contrastive objective aligns short synchronized audio and video segments, helping the two modalities understand each other. Meanwhile, semantic distillation preserves meaningful modality-specific information by distilling features from pretrained semantic encoders into each latent space. As the authors explain, "contrastive learning helps audio and video understand each other, while distillation helps each modality express itself clearly."
Extensive experiments demonstrate that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation tasks. Notably, these gains come with minimal impact on reconstruction performance.
The findings underscore the importance of learning unified multimodal representations as a foundation for omnimodal modeling. The researchers hope this work encourages further exploration of unified representations for generation, interaction, and virtual agents.