Microsoft researchers have introduced Mage-VL, a codec-native streaming multimodal foundation model designed to address Moravec's paradox in vision-language models (VLMs). While standard VLMs excel at complex offline visual reasoning, they struggle with simple streaming perception tasks and are computationally inefficient. Mage-VL aims to bridge this gap with a custom tokenizer and dual-system architecture.
The core innovation is Mage-ViT, a tokenizer that replaces uniform frame sampling with selective encoding of dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16x16 patch level, this approach reduces visual token consumption by over 75% while preserving spatiotemporal context. Mage-ViT was trained from scratch on approximately 560 million unlabeled images and 100 million unlabeled video frames, matching or outperforming flagship encoders trained on billions of image-text pairs.
Mage-VL employs a bio-inspired dual-system architecture: a lightweight System 1 event gate for proactive streaming perception and a causal System 2 decoder for deeper reasoning. This design enables efficient real-time multimodal understanding and interaction. The model also leverages AI4AI data pipelines, including prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes.
Extensive evaluations show that the 4B-parameter Mage-VL matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup. It comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. The paper also delivers seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.