Video large language models (VideoLLMs) pair video representations with pretrained large language models and condition their output on a text prompt. According to a new survey posted on Hugging Face, these systems perform strongly on captioning, question answering, retrieval and temporal grounding, but their computation and memory costs grow with frame count and context length, which limits deployment in real-time, mobile and resource-constrained environments.
The survey examines inference-efficiency mechanisms that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token counts. It analyzes bottlenecks across four pipeline stages: frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. Methods are organized by the stage at which they act, covering VideoLLMs developed since late 2022 alongside earlier frame-sampling and vision-encoder techniques that remain part of current pipelines.
In total, the authors review 125 papers and only include methods that report concrete efficiency gains. Where papers share a host model and input protocol, the survey assembles accuracy–cost comparisons and keeps them separate from heterogeneous cross-paper numbers, which the authors note are not directly comparable.
Two gaps stand out: audiovisual efficiency is barely studied, and efficiency evaluation is not standardized. The authors maintain a living repository of the reviewed papers at github.com/momentslab/awesome-efficient-videollm.