Visual-token pruning is a key technique for reducing the inference cost of multimodal large language models (MLLMs), but existing methods rely on fixed, handcrafted heuristics and costly expert trial and error. As models and objectives diversify, manually exploring the design space becomes increasingly difficult.
In a new paper, researchers from Hugging Face propose AutoPrune, a training-free framework that leverages large language models (LLMs) to automatically design effective visual-token pruning policies. The core innovation is a Token Pruning Domain-Specific Language (TPDSL) with 131 reusable atoms for budget control, token scoring, selection constraints, and token reassembly.
A key property of TPDSL is its residual formulation: each search state is represented as a modification of a strong base policy. This narrows the search space and focuses the LLM's attention on the most consequential policy components, making the search more efficient and effective.
Experiments on 14 multimodal benchmarks and three MLLM backbones demonstrate AutoPrune's effectiveness, efficiency, and transferability. Notably, even when removing 94.4% of visual tokens, AutoPrune preserves more than 99% of full-token performance while reducing FLOPs by 9.9x and prefill latency by 6.4x.
The paper is available on Hugging Face and was highlighted by the Librarian Bot, which also recommended related works such as TOPS, DIVE, and AnchorPrune.