Researchers have introduced MassAlloc Attention (MALA), a fused attention primitive that keeps score access to every legal causal interaction but decides how much post-score computation each region deserves. The work is presented alongside a companion method, CoWindow Attention (CoWA), in a Hugging Face paper listing.
The motivation is that FullAttn often assigns negligible normalized mass to much of the causal score space, yet dense kernels still execute the complete post-score path after forming each QK tile. MALA instead uses normalized contribution to allocate that computation: the forward pass relies on its evolving online-softmax normalizer, while the backward pass reuses the finalized normalizer to derive a nested retained support using only standard attention state. A common tolerance governs both training and inference.
In a matched-work study at 8K context, MALA approached a per-instance reference-mass oracle under exactly matched total post-score work, with mean omitted mass of 0.0188% versus 0.0182% for the oracle. Across context lengths from 1K to 32K tokens, the same tolerance maintained low output and gradient errors relative to the reference. In a controlled associative-recall comparison, MALA reached 89.67% accuracy at 8K versus 89.97% for FullAttn.
An attention-operator benchmark at 128K tokens with tensor parallelism on 8×H100 (TP=8) reported 2.2x forward, 3.0x backward, and 1.6x decoding speedups for MALA relative to FullAttn. The companion CoWA method reported 7.4x forward, 8.6x backward, and 3.0x decoding speedups in the same setting.
Scaling-law training from 0.6B to 14B parameters showed MALA closely tracking FullAttn in perplexity while reducing total training FLOPs. During 14B training with 32K context, MALA reduced total training FLOPs by 23.1% (CoWA by 28.5%). The resulting 14B models and 32B models from separate continued training achieved comparable knowledge, reasoning, and long-context retrieval scores to FullAttn.
The authors frame the broader goal as identifying which attention computations can be eliminated and turning those savings into practical ML infrastructure gains, from method design through kernel implementation to model training.