In a new research paper, Hugging Face researchers delve into the 'physics' of multimodal pretraining, aiming to demystify how different modalities—such as language, vision, and visual generation—interact during unified training. The study, released on the Hugging Face platform, provides empirical clarity through systematic experiments on both synthetic and large-scale real-world datasets.
The paper identifies four key insights. First, it disentangles 'knowledge flow,' revealing distinct patterns of influence and asymmetry in how knowledge transfers across modalities. Second, it shows that data complexity largely determines whether modalities are synergistic or competitive, and highlights architectural choices—like shared attention and normalization with modality-specific feed-forward layers—that promote synergy. These behaviors generalize across different visual tokenizer designs.
Third, the research advocates for 'early unification,' demonstrating that training modalities jointly from the very beginning is more effective than late alignment or sequential training. This process uncovers a 'vision laziness' phenomenon, where delayed integration leads models to rely on language priors. Finally, the authors derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget.
These findings are validated at scale by training multiple 13.5B parameter mixture-of-experts (MoE) models on 2 trillion tokens. The study aims to provide a principled foundation for understanding and scaling multimodal pretraining, offering practical guidance for researchers and engineers in the field.