Researchers from NVIDIA and collaborating institutions have introduced Sol-Attn, a training-free sparse attention mechanism that accelerates video generation inference by dynamically sparsifying attention computations. The method addresses a key bottleneck in diffusion transformers, where long token sequences make attention the dominant cost.
Existing training-free sparse attention methods suffer from rigid routing and lossy sparsification. Sol-Attn overcomes these limitations by unifying dynamic routing, sparse computation, and approximation correction in a single online-softmax pass. Its core innovation is on-the-fly block thresholding with proxy-score reuse: it selects critical key-value blocks by comparing proxy scores against a threshold during online softmax, enabling dynamic yet controllable budgets without materializing the proxy map. Unselected blocks are not discarded entirely; their proxy scores are reused to approximate their contribution.
Experiments on image and video generation tasks show that Sol-Attn advances the quality-efficiency frontier. Against dense FlashAttention-3, it achieves 2.02× end-to-end speedup on Wan 2.1-14B, 2.12× on HunyuanVideo-13B, and up to 2.4× on LTX 2.3. When integrated into Sol-Engine (with kernel fusion and caching), speedups reach 3.48× on Wan 2.1-14B and 5.08× on HunyuanVideo-13B.
The method is already available in Sol-Engine. The paper and code are publicly accessible, with the B200 kernel still under further optimization.