Vision encoders are essential for vision-language models, but scaling them densely increases compute and latency. A new paper from Hugging Face researchers proposes MoE-ViE (Mixture-of-Experts Vision Encoder), a family of models that leverage fine-grained MoE topologies to achieve superior performance without the usual overhead.
The study systematically explores MoE designs for CLIP-style vision encoders, finding that fine-grained topologies significantly outperform both dense and standard MoE variants. To further improve efficiency, the authors introduce an auxiliary-loss-free balancing method for better expert utilization and a specialized MoE kernel that reduces inference latency.
For video understanding, MoE-ViE uses frame-level distillation with a novel freezing mechanism, preserving image knowledge while enhancing video capabilities. The models are pretrained across various sizes, consistently outperforming their dense counterparts.
Notably, the largest MoE-ViE model matches the zero-shot performance of a state-of-the-art encoder that is 1.7x its size, while running at 76% of the latency. When aligned with a language model, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters.
The code is available on GitHub at facebookresearch/moe_vie.