Sparse attention is one of the main techniques used to cut the latency of diffusion transformers in long-sequence generation tasks such as video and high-resolution 3D asset generation. However, existing sparse methods can degrade generation quality and fidelity when sparsity levels are pushed high.
In a new paper, researchers use controlled oracle comparisons to trace this degradation to three distinct sources: constraints imposed by token grouping, inaccurate interaction selection, and the attention contributions lost when tokens are discarded.
Guided by that analysis, the authors propose Meta-Cached Sparse Attention (MC-Sparse), a training-free framework that selects individual key-value (KV) tokens while organizing similar queries into tile-aligned groups for efficient GPU execution. MC-Sparse caches metadata — query groups, KV indices chosen using exact attention probabilities, and residuals between dense and sparse attention outputs — and reuses it across subsequent denoising steps.
Across video and 3D generation models, MC-Sparse reportedly achieves higher fidelity to dense-attention outputs and larger denoising speedups than existing sparse-attention baselines, without visible quality degradation.
Relative to dense attention, the method delivers a 1.80x denoising speedup on Minimax-H3-Base and a 2.32x speedup on 3D asset generation, both with negligible quality loss, according to the paper.