Efficient long-video understanding with vision-language models (VLMs) is often framed as selecting informative frames or visual tokens at a fixed native resolution. A new paper on Hugging Face challenges that framing, showing that per-frame resolution can instead be traded for denser temporal coverage.
The authors report that front-end decoding latency depends on the size of the candidate pool rather than the final token budget. Their empirical study across multiple VLMs and long-video benchmarks yields three findings: dense low-resolution sampling outperforms sparse native-resolution sampling at matched token budgets; resolution-sensitive tasks benefit from selected high-resolution frames; and front-end decoding dominates wall time for hour-long videos.
Motivated by these findings, the paper introduces LoHi, a training-free, single-pass framework that combines a dense low-resolution video stream with sparse high-resolution image frames through the VLM's native video and image pathways. LoHi-Anchor selects high-resolution frames using codec I-frame metadata, while LoHi-SemDiv uses query relevance and visual diversity over CLIP features.
Across three long-video benchmarks, LoHi improves average accuracy by 10.6 percentage points over the native-resolution baseline at a matched token budget and by 5.2 percentage points over the strongest prior efficiency method. It also reduces front-end decoding latency by up to 7x on hour-long videos.
The paper also notes limits of keyframe selection and token pruning: both stay at native resolution and discard afterward, decoding 256 full frames only to keep 16 (or drop roughly 94% of tokens), paying the decoding cost while losing temporal coverage. A plain low-resolution baseline beats both (64.4 vs. 60.8 / 59.3 on VideoMME). LoHi decodes 128 frames once, feeding dense low-resolution video plus a few high-resolution images, and reaches 66.8 on VideoMME with 3.9s time-to-first-token (vs. 6.0–9.0s) on Qwen3-VL-4B.