Long-video language models face a fundamental constraint: they cannot process every frame of a video. An hour of footage at one frame per second yields 3,600 images, yet most systems only keep a small, fixed subset. A new paper from Hugging Face, titled "Select, Compress, Reinvest," investigates whether the choice of which frames to keep is merely a preprocessing detail or a critical factor in model accuracy.
The researchers designed a controlled experiment that isolates three variables: frame selection, spatial compression, and reinvestment of the token savings. They tested six training-free selection rules across three long-video benchmarks and two answering models, holding all other factors constant. This approach addresses a common problem in the field: published selectors often change multiple components at once, making fair comparisons difficult.
The results are striking. Selection proved to be the largest single lever for accuracy. On LongVideoBench's hour-long bin, selecting eight frames based on query relevance outperformed sixteen uniformly spaced frames by 6.9 points. Remarkably, Orthogonal Matching Pursuit (OMP), a decades-old sparse-approximation algorithm used without modification, matched or came within one point of every purpose-built selector across all benchmarks.
Spatial compression, in contrast, was nearly free: halving each frame's spatial budget at fixed timestamps cost at most 0.44 points. However, the real benefit emerged when the freed tokens were reinvested into more frames. Spending the savings on twice as many compressed frames, at a measured cost no higher than the original eight, returned an additional two to three points of accuracy. The authors emphasize that compression only pays off when its savings are spent this way.
The study also highlights the need for a unified evaluation harness. An implementation bug in their own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget illustrate why cross-paper comparisons can be misleading. The authors advocate for controlled, single-harness evaluations to fairly assess frame selection strategies.