Vision-language models (VLMs) have become increasingly capable, but their computational cost is dominated by vision tokens, which take up most of the input sequence. Compressing those tokens has therefore become a central route to faster inference. However, most existing compression methods were designed for softmax attention and do not transfer cleanly to emerging hybrid architectures that use linear attention, such as Qwen3.5.
According to a new paper posted on Hugging Face, both attention-based and similarity-based compression approaches degrade noticeably in this regime, creating a need for methods tailored to linear attention. The authors introduce V-CoLA, a training-free vision token compression framework built specifically for that setting.
V-CoLA scores each vision token directly from the recurrent state using a uniqueness-aware importance criterion. This criterion combines how much a token contributes to the final state with how much it changes the state in the short term. Tokens are then merged adaptively in chunks shaped by where importance is concentrated, and redundant vision tokens are dropped in deeper layers. All components are optimized at the implementation level to stay compatible with the chunk-wise parallelism of linear attention.
Experiments across multiple benchmarks report strong results: V-CoLA retains 99.5% of the original performance while using only 50.0% of vision tokens, and still exceeds 88.0% with as few as 12.5% of tokens. The method also delivers a prefill speedup of 1.86× to 6.15×.
The paper is accompanied by automated recommendations from Hugging Face's Librarian Bot, which surfaced related 2026 work on visual token pruning and KV cache compression, including DIPrune, VisCache, PACE, SinkPruner, and HAP.