Researchers have introduced TempCloze, a video cloze benchmark designed to evaluate visual temporal reasoning in Video-LLMs while reducing reliance on linguistic shortcuts. According to the paper, existing temporal reasoning benchmarks are often mediated by language, leaving room for shortcuts from option wording, answer correlations, or language priors.
In TempCloze, models receive the beginning and ending clips of a video and must identify the true missing middle from four candidates. The benchmark contains 1,521 carefully filtered videos drawn from seven sources, mainly long-take and egocentric footage.
Distractors are constructed from the same source along three dimensions: Semantic (what event should happen), Alignment (when it should occur), and Progression (how it should unfold). Shared scenes and objects are used to reduce appearance cues.
Evaluating 10 proprietary and 21 open-source Video-LLMs, the authors identify Alignment as the primary bottleneck: models often recognize plausible semantic content and local event progression but struggle with temporal alignment.
The paper also reports error pattern and behavioral sensitivity analyses on TempCloze-Mixed and TempCloze-Hard with four representative models, examining where errors arise and how candidate order, context direction, visible span, frame density, and test-time scaling influence model choices.