Multimodal large language models (MLLMs) are moving toward continuous audio-visual reasoning, and that progress has created an urgent need for evaluations capable of exposing their capability limits. Audio-visual captioning, according to a new research paper, is an ideal diagnostic task for this purpose.
Existing benchmarks, however, face what the authors describe as a coupled trade-off. Whole-caption scores offer coverage but no localization, local probes offer localization but no coverage, and unconstrained LLM judges introduce instability into the scoring process.
To address this, the researchers introduce OmniCapBench (Omni-Video Caption Benchmark), which reframes audio-visual caption evaluation as a deep-structured diagnostic framework. Instead of predicting free-form text, models are evaluated against sets of atomic, verifiable evaluation units organized into three tracks: entity references, visual shots, and audio events.
This structured approach enables reliable scoring through deterministic constraint checks combined with localized LLM-based semantic comparisons. The benchmark comprises 786 densely annotated videos.
According to the paper, OmniCapBench can distinguish specific MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier models revealed strong local perception but weak long-horizon audio-visual reasoning, particularly in identity drift and cross-modal misalignment — findings the authors present as a fine-grained roadmap for omnimodal development.