Researchers from Hugging Face have proposed VisCo, a novel framework for visual token compression in vision-language models (VLMs). The work addresses the high inference latency and memory overhead caused by the large number of visual tokens processed by VLMs.
Existing approaches fall into two categories: training-free strategies that rely on heuristic metrics and suffer performance degradation at high compression ratios, and training-based methods that introduce external compression modules, requiring costly retraining and potentially compromising the VLM's pretrained priors. VisCo takes a different approach by reusing the pretrained VLM itself as an intrinsic compressor.
VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. This design eliminates the need for external modules and minimizes retraining cost while preserving the VLM's original capabilities.
Experiments demonstrate that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression. Notably, it remains stable even in the extreme single-token setting. Furthermore, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression.
The code is available on GitHub at https://github.com/Zyvpeng/VisCo.