Long visual contexts remain a significant hurdle for vision-language models (VLMs), as performance degrades with an increasing number of distractors and processing all tokens simultaneously becomes computationally prohibitive under GPU memory limits. A new paper introduces ReToken, a method that tackles this challenge with a single learnable embedding designed as an explicit retrieval target.
ReToken operates by selecting a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. This approach avoids the need to process the entire visual context at once, making it both efficient and effective. The method is trained on only a small image-QA dataset, yet it delivers consistent gains across multiple benchmarks.
On the Visual Haystacks benchmark, ReToken improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points, representing over 20% relative improvement. For video, the method transfers zero-shot to long-video tasks on LVBench, yielding an 8.0-point gain with Qwen3VL-8B.
Thanks to its lightweight design, both training and long-video inference can be performed on a single H100 GPU. The code is publicly available on GitHub, enabling further research and adoption.