Most modern multimodal large language models (MLLMs) rely on a pretrained visual encoder to supply a strong visual prior. An alternative, encoder-free approach learns visual representations directly from raw pixels, offering a simpler and more unified architecture. However, its scaling behavior had not been systematically studied—until now.
Researchers compared scaling laws for encoder-free and encoder-based MLLMs and reported three main findings. First, removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while the allocation for text remains nearly unchanged.
Second, the two architectures show nearly overlapping loss–compute frontiers on the text objective, but diverge on the multimodal objective. Encoder-free models underperform at small scales, yet are predicted to catch up at around 10^22 FLOPs—well within practical pretraining budgets.
Third, without a visual encoder, the language model learns to take over its role through vision-specific adaptation. Bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated.
Overall, the results indicate that the advantage of the visual prior from a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.