Researchers at Hugging Face have released VGI-Bench, a new benchmark designed to probe the visual intelligence of video generation models beyond mere perceptual quality. The benchmark aims to assess whether these models can reason through evolving visual processes, not just produce plausible frames.
VGI-Bench comprises 27 tasks and 810 instances, organized under a two-level taxonomy of task domains and skill tags. This structure enables fine-grained evaluation of various reasoning capabilities, from basic object tracking to complex causal inference.
In their evaluations, the team found that current generative systems can solve a subset of visually grounded reasoning tasks, but they remain far from reliable. Even the strongest model tested, Seedance 2.0, achieved only 51.0% accuracy under the benchmark's criteria.
The analysis also explored output failure modes, sensitivity to input conditions, and the transfer boundary from synthetic fine-tuning. A key insight from internal denoising analysis is that models exhibit limited self-correction: later denoising steps tend to refine early hypotheses rather than correct fundamental reasoning errors.
The researchers hope VGI-Bench will stimulate the development of next-generation video generation models with more robust visual reasoning abilities. The benchmark is publicly available at the project website.