Researchers at Hugging Face have released PerceptionBench, a new benchmark designed to isolate and evaluate the atomic visual perception abilities of Multimodal Large Language Models (MLLMs). The work addresses a critical flaw in existing evaluations: they often conflate perception errors with failures in reasoning or domain knowledge, making it difficult to pinpoint where models truly struggle.
The team adopted a bottom-up approach by analyzing the earliest failure points in responses from frontier MLLMs across 42 existing benchmarks. This analysis yielded an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Using this taxonomy, they constructed 3,000 verified questions, each isolating a single capability with short, unambiguous answers. Difficulty is derived purely from perception, not reasoning or knowledge.
Results from testing sixteen frontier MLLMs reveal that atomic visual perception remains largely unsolved. No model achieved more than 60% accuracy, and perception-related hallucination emerged as the weakest capability on average. Notably, models with similar overall scores often exhibited sharply divergent capability profiles, underscoring the need for fine-grained diagnostic tools.
PerceptionBench provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs, offering a clearer path toward improving their foundational visual understanding.