Language models often spend most of their attention on a small fraction of the context, yet they read the entire KV cache to find the few tokens that matter. This inefficiency becomes critical in long-context scenarios, such as a 1M-token conversation, where global attention layers must scan the full context to generate each token of a reply.
To address this, researchers at Hugging Face introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought. The model partitions generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read.
In zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding by 52.0% and 31.1%, respectively, with modest accuracy drops of 1.27 and 2.75 percentage points. Notably, these accuracy drops shrink with model scale.
The authors note that DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore. The paper is available on Hugging Face's research platform.