Researchers have proposed CoWindow Attention (CoWA), a structured attention architecture that distributes access to the causal history across KV heads rather than exposing the complete history to every head as FullAttn does. The work is presented in a paper titled CoWindow Attention: Full Causal Coverage Is a Collective Property.
In CoWA, all heads share near-diagonal and prefix-sink windows, while complementary long-range windows partition the remaining history. Their union provides full causal coverage even though each head attends sparsely to distant tokens. The pattern is position-defined, requires no learned router or indexer, is used consistently during training and inference, and aligns with KV-head tensor parallelism.
A window-matched ablation at 8K isolates the effect of complementary long-range allocation: CoWA with 100% collective coverage reaches 89.73% accuracy, compared with 89.97% for FullAttn, while duplicated long-range windows perform substantially worse. In a broader controlled associative-recall comparison with matched token budgets, CoWA closely tracks FullAttn as context grows, whereas other sparse patterns lose a substantial fraction of the associations.
In an attention-operator benchmark at 128K tokens with tensor parallelism, CoWA reduces forward and backward latency during training by 7.4x and 8.6x and decoding latency during inference by 3.0x over FullAttn. Its per-rank peak operator memory matches FullAttn during training and is 7.6x lower during decoding.
Across scaling-law training from 0.6B to 14B parameters, CoWA closely tracks FullAttn in perplexity while reducing total training FLOPs. The resulting 14B models and 32B models from separate continued training achieve comparable knowledge, reasoning, and long-context retrieval scores to FullAttn. The authors also describe a companion approach, MassAlloc Attention (MALA), which preserves full causal QK scoring and uses attention's own softmax statistics to reduce subsequent computation in low-contribution regions.
The team reports that during 14B training with 32K context, CoWA and MALA reduced total training FLOPs by 28.5% and 23.1% respectively while maintaining performance comparable to FullAttn on evaluated capabilities. The results suggest that full causal coverage can be a collective property of the head ensemble rather than a duplicated property of every head.