Researchers from Hugging Face have unveiled a new approach to spatio-temporal video grounding (STVG) that promises to dramatically speed up the process. Their paper, titled "Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding," introduces Parallel Tube Decoding (PTD), a generative framework that decodes spatial and temporal grounding simultaneously.
Traditional multimodal large language models handle STVG by serializing dense localization trajectories autoregressively. This causes decoding latency to grow with the length of the tube (the sequence of frames) and allows errors to propagate over time. PTD breaks this pattern by decomposing grounding into a temporal block followed by time-conditioned spatial blocks that are decoded in parallel. This removes both token-level and trajectory-level dependencies, reducing sequential decoding depth to a fixed two rounds regardless of tube length.
To enable parallel spatial generation, the team introduced Decoupled Block Attention, which preserves shared video-query context while eliminating cross-box dependencies. They also employed localization-aware policy optimization to refine temporal boundaries and spatial geometry.
The results are striking: on the VidSTG benchmark, PTD reduces tube completion latency by 79 times and increases spatial decoding throughput by 92 times compared to standard autoregressive decoding, all while improving grounding accuracy. Even with a compact 4B parameter backbone, the model performs favorably on VidSTG and HC-STVG, and generalizes zero-shot to temporal grounding, grounded VideoQA, and referring video object tracking.
The authors conclude that parallel tube generation is an efficient and effective alternative to autoregressive localization in video understanding tasks.