Researchers at Hugging Face have published a study titled 'Scaling Native Multimodal Pre-Training From Scratch,' exploring how to optimally scale vision-language models trained from scratch on multimodal data. The work addresses a key gap in understanding the scaling properties of this paradigm, which promises deeper cross-modal integration compared to traditional late-fusion approaches.
The team trained transformer-based vision-language models under fixed computational budgets, varying model size and token count. They found that minimal objective loss follows a predictable compute law, while compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives exhibit different scaling behaviors.
Key findings include that the language allocation law is largely invariant to data composition, meaning language learning remains stable regardless of the multimodal data ratio. In contrast, the multimodal allocation law is highly sensitive to data composition: text-heavy mixtures become compute-efficient only at larger model scales, shifting optimal resource allocation toward greater model capacity.
By modeling the influence of data composition on compute laws and allocation exponents, the researchers derived an efficiency frontier that specifies precise configurations of model size, token count, and data mixture. Downstream evaluations show that native multimodal pre-training induces positive cross-modal transfer, enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning.
This empirical research establishes foundational groundwork for predictably scaling multimodal foundation models, offering practical guidelines for efficient training.