Linear attention has become a staple of frontier language models because it enables efficient long-context inference and constant-memory decoding. But its fixed-size recurrent memory forces an online decision at every token: what to write, and how strongly to overwrite existing associations, before the model knows which information future queries will need.
Delta-rule models learn that write strength from the current token embedding, yet they do not track confidence in the memory estimate, so each write cannot adapt to accumulated evidence. A new paper, Kalman Delta Networks: Uncertainty-aware Associative Memory, addresses this gap by reformulating recurrent associative memory as a linear-Gaussian state-space model, for which the Kalman filter is the optimal recursive estimator.
The resulting family of models, Kalman Delta Networks (KDNs), propagates both the memory state and its uncertainty through the transition. This lets the Kalman gain weight each residual write according to accumulated evidence and observation reliability. Under this formulation, Delta-style updates emerge as a special case that substitutes a token-wise isotropic surrogate for predictive covariance and omits covariance tracking.
Exact uncertainty tracking, however, requires a dense, state-dependent Riccati recursion that is poorly suited to GPU-parallel linear-attention scans. The authors therefore introduce two scan-compatible approximations: Diagonal KDN, which projects each one-step posterior onto the diagonal Gaussian family via online mean-field variational inference, and Isotropic KDN, which uses an isotropic approximation with a single uncertainty scalar per head. Their uncertainty recurrences are Möbius maps, enabling associative scans with logarithmic parallel depth.
Across controlled pretraining at 750M and 1.3B parameters, the KDN variants consistently improved perplexity and mean downstream accuracy over state-of-the-art linear-attention models, according to the paper. The work connects classical Kalman filtering with modern linear attention while keeping the architecture efficient for large-scale language modeling.