Moonshot AI's Kimi has released WorldVQA, a new benchmark designed to measure the factual correctness of Multimodal Large Language Models (MLLMs) regarding visual world knowledge. The benchmark addresses a critical question: whether models truly recognize specific entities or merely hallucinate based on visual patterns.
WorldVQA comprises 3,500 high-quality image-question pairs, each with a single, verifiable ground-truth answer. The dataset spans nine categories and explicitly separates data into Head (common knowledge) and Tail (rare/long-tail knowledge) to assess performance degradation as knowledge becomes more obscure. All pairs underwent rigorous multi-stage human verification to ensure quality.
Results show that even state-of-the-art models struggle, often achieving below 50% accuracy on long-tail visual knowledge. The benchmark also evaluates calibration—the alignment between model confidence and actual accuracy—using Expected Calibration Error (ECE) and Slope metrics. Kimi-K2.5 achieved the best performance with an ECE of 37.9% and a Slope of 0.550, but all models exhibited overconfidence, concentrating predictions in the 90-100% confidence range.
WorldVQA is open-sourced, with the dataset and evaluation scripts available on GitHub and Hugging Face. The team hopes it will drive progress toward more factually reliable multimodal AI.