Moonshot AI has introduced WorldVQA, a new benchmark aimed at evaluating the factual correctness of Multimodal Large Language Models (MLLMs) regarding visual world knowledge. The benchmark addresses a critical question: whether models truly recognize specific entities or merely hallucinate based on visual patterns.
The dataset comprises 3,500 high-quality image-question pairs, designed to test encyclopedic breadth across the world. It features three core design principles: factuality and unambiguity (each question has a single verifiable answer), a rich taxonomy spanning nine categories, and a head vs. tail distribution to measure performance degradation on rare knowledge.
Initial results show that even state-of-the-art models struggle, often falling below 50% accuracy on long-tail visual knowledge. The benchmark also reveals a universal tendency toward overconfidence among all evaluated models, with Kimi-K2.5 achieving the best calibration metrics but still far from ideal.
WorldVQA is open-sourced, with the dataset and evaluation scripts available for the community to address the visual knowledge gap. The paper is available on arXiv, and code and data are hosted on GitHub and Hugging Face.