Diffusion Transformers have become the leading approach for video generation, but their long spatiotemporal sequences make attention the dominant cost in deployment. A practical low-bit attention kernel must be both accurate and fast, and a new paper from Hugging Face introduces VC-Attention to tackle both constraints.
Accuracy in low-bit attention is limited by outliers: a block's quantization scale is set by its largest entries, leaving typical values confined to a narrow representable range. While prior work smooths queries and keys, value outliers follow no fixed channel or spatiotemporal structure and remain the main source of output error. VC-Attention's V-Smooth component reorders value tokens through lightweight online clustering so tokens in a hardware block quantize well together. It quantizes only the residual after subtracting the block mean and restores that mean from the row sum the online softmax already maintains.
Speed is limited by softmax: low-bit Tensor Cores accelerate only the two matrix multiplications, so the high-precision exponential between them becomes the longest pipeline stage on datacenter GPUs. The paper's ExpCast-FP8 maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion.
The authors implemented VC-Attention for B200, B300, H200, RTX PRO 6000, and RTX 5090. Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, the method improves fidelity over low-bit baselines, speeds up the attention kernel over BF16 FlashAttention-4 by 1.46-1.59x on datacenter Blackwell and Hopper and by 2.3-3.6x on workstation cards, and generates clips 1.13-1.19x and 1.36-1.70x faster end to end.
According to the authors, VC-Attention requires no retraining and works with existing sparse attention methods. On MiniMax-H3, it offers a 1.6x attention speedup on B200 and 1.5x on B300 over BF16 FlashAttention-4. The work is positioned as a step toward faster, more affordable multimodal inference.