Moonshot AI has unveiled PerceptionBench, a novel benchmark designed to isolate and evaluate visual perception in multimodal large language models (MLLMs). Unlike traditional benchmarks that mix perception with reasoning and knowledge, PerceptionBench focuses purely on atomic perceptual capabilities, derived from analyzing how current models fail across 42 existing benchmarks.
The benchmark identifies ten atomic perceptual categories: Visual Relation, Counting, Attribute, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination. Each of the 3,000 verified questions in the released set is crafted to test a single capability, requiring only visual inspection without any reasoning or external knowledge.
Initial results are striking: across sixteen frontier MLLMs, none achieves 60% accuracy, and perception-related hallucination emerges as the weakest capability on average. Notably, models with nearly identical overall scores can differ dramatically in what they actually perceive, underscoring the need for capability-level diagnostics.
PerceptionBench is built on a failure-driven taxonomy, where each category is discovered from real model failures, attributed to the earliest erroneous step. The dataset includes 17,000+ verified questions in-house, with the released 3,000 subsampled to balance categories and difficulty. The authors emphasize that the benchmark's difficulty stems from perception itself, not reasoning or knowledge.
Moonshot AI is open-sourcing the dataset and evaluation code to help the community address the visual perception gap. The benchmark aims to drive progress toward multimodal AI that sees faithfully and consistently.