Hugging Face researchers have introduced NeoMME, a new family of multimodal and multilingual encoders designed to address the computational overhead of using generative vision-language models for non-generative tasks like visual document retrieval. NeoMME models, available in 260M and 800M parameter sizes, process multilingual text and raw image patches in a single bidirectional Transformer encoder, eliminating the need for separately pretrained vision encoders or causal language models.
The models are pretrained from scratch using a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both variants support a 16,384-token context, sufficient to encode up to two standard 4K UHD images. This design aims to provide a more efficient foundation for tasks that require understanding both text and visual content without the generative overhead.
To demonstrate downstream capabilities, the team fine-tuned NeoMME with jointly trained dense and late-interaction heads for visual document retrieval. On the ViDoRe v3 benchmark, the NeoMME-Retriever 260M achieved an nDCG@10 of 0.523, outperforming all evaluated models strictly below 800M parameters, while the 800M version reached 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with approximately twice the throughput of ColModernVBERT.
A key innovation is the compression of late-interaction multimodal document embeddings. Through hierarchical token pooling and asymmetric quantization, the embeddings are compressed by 255x while preserving over 95% of the baseline nDCG@10. This could significantly reduce storage and computational costs in retrieval systems.
NeoMME has been integrated into Hugging Face Transformers, and the pretrained backbones and retrieval-compatible checkpoints are released under the Apache 2.0 license. The models are available at the provided Hugging Face collection link.