Multimodal large language models (MLLMs) have become adept at general visual tasks such as visual question answering, but a new benchmark suggests they remain surprisingly weak at a fundamental comparative skill: spotting what has changed between two similar images.
Researchers introduced VDiff-Bench, a challenging multiple-choice benchmark for fine-grained image difference identification. It contains 1,756 four-way questions over image pairs and spans 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination.
Each question pairs two image inputs with four choices: the true difference, two hard negative descriptions, and a "no difference" distractor. The negatives are specifically curated to be ground-truth-conditioned, forcing models to distinguish the actual change from nearby semantic alternatives rather than relying on superficial cues.
Testing 11 state-of-the-art open- and closed-source MLLMs showed that fine-grained visual comparison remains brittle. Performance was uneven across sources and change categories, with persistent failures on subtle low-level changes such as noise and texture. Three open-source models in the 7–8B range scored 52.5–70.6% on semantic changes but only 8.7–33.3% on low-level changes like noise and texture, often falsely assuming no change had occurred.
Notably, despite the strong overall performance of other closed-source commercial models, Grok 4.3 showed a remarkable drop on noise and texture differences, falling significantly behind large open-source models such as Kimi K2.5 and K3.
The authors position VDiff-Bench as a targeted diagnostic for comparative visual understanding in MLLMs, exposing failures that standard single-image vision-language tasks do not capture. A project page is available for further exploration.