Large Language Models (LLMs) suffer from quadratic attention complexity, leading to high memory and energy consumption. As each new token is processed, keys and values must be stored indefinitely, making scaling unsustainable. Linear attention has been proposed as a fix, promising low-cost, state-of-the-art performance, but it has not been properly compared to simpler baselines.
In a new paper, researchers at Hugging Face demonstrate that Sliding Window Attention (SWA) with sinks performs as well as or better than post-trained linear attention models across multiple LLMs and downstream tasks. On long-context reasoning benchmarks like Needle-in-a-Haystack and BABILong, SWA achieves massively higher performance—2 to 10 times higher than linear attention.
SWA requires no post-training, is extremely fast, and uses low memory, making it a cheap and reliable inference solution. The authors strongly recommend switching to SWA instead of post-training linear models to reduce inference memory costs. They note that linear attention models may show promise but likely need to be trained from scratch or undergo extensive post-training to even match SWA.