Omni-modal Large Language Models (OmniLLMs) that process text, audio, and video simultaneously face significant memory and inference costs due to long token sequences. Existing compression methods typically select important tokens under fixed budgets, but the preceding problem of how to allocate budgets across modalities remains underexplored.
Researchers from Hugging Face introduce OmniDelta, a training-free, skill-driven framework that couples intent-aware inter-modal allocation with content-aware intra-modal allocation. The method constructs audio and video skill pools to shift the fixed retained-token budget according to query demand, then reallocates modality budgets over audio segments and video frames using local complexity and temporal redundancy.
OmniDelta can be combined with existing pruning strategies, preserving the total retained-token ratio while changing where the budget is spent. Experiments on four audio-video benchmarks with two Qwen2.5-Omni models show that OmniDelta establishes a new accuracy-efficiency Pareto frontier across pruning ratios.
At 25% token retention on Qwen2.5-Omni-7B, OmniDelta reduces GPU memory by 22.0% and achieves a 1.64x end-to-end speedup over full-token inference.