Text-rich images, such as documents and screenshots, require AI models that can read, retrieve, and compress language directly in pixel space. However, existing pixel-text encoders often struggle with fixed-resolution pretraining, visual shortcut learning, weak visual grounding, and limited multilingual understanding.
In a new research paper, Hugging Face scientists systematically investigate the fundamental design principles for robust visual text representation learning. Through controlled ablations, they identify four critical components: variable image resolutions and rendered font sizes, natural image-text grounding, layout-aware rendering, and a two-stage multilingual curriculum.
Based on these principles, the team trained Pixel Linguist II, a native-resolution vision encoder trained on 280M examples with on-the-fly rendering and unified contrastive grounding. The model sets new state-of-the-art results on English, cross-lingual, and multilingual Visual STS and ViDoRe benchmarks.
Notably, Pixel Linguist II shows strong compression robustness: its representations remain effective even when 80% of visual tokens are compressed. With only 40% of tokens retained, it still outperforms the full-budget Qwen2.5-ViT on multimodal LLM downstream tasks, improving average performance by 2.75%.
The code and resources are publicly available on GitHub.