Researchers have released a paper on Hugging Face titled "Fast Weight Attention for Continual Learning," which analyzes recurrent fast-weight memories and selective state-space models as online learning rules under autoregressive semantics. The work aims to address the challenge of compressing an expanding context into a fixed-size recurrent state, effectively treating state transitions as learning processes.
The paper introduces a family of normalized first-order updates for two objectives: squared-error regression and negative inner-product. The regression family includes Falcon-1 (scalar NLMS), Falcon-2 (per-column extension), and Falcon-3 (sliding-window mini-batch), with inner-product variants Falcon-1A/2A/3A. These updates come in recurrent, masked-parallel, and chunk-parallel forms, along with numerically stable positive-decay renormalization.
Key findings show that representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition tasks. The framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models, offering a structured approach to continual learning.
The paper also highlights related works recommended by the Semantic Scholar API, including Maglev, MARCH, and DART, which explore similar themes in recurrent memory and long-context modeling.