Hugging Face researchers have introduced Maglev, a novel recurrent Transformer architecture designed to handle long contexts efficiently. The model uses a fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training, addressing key limitations of previous approaches.
Maglev consists of two coupled models: a prefiller (Q) that leverages full attention to produce memory targets, and a decoder (P) that uses only sliding-window attention and recurrent key/value injection to generate decoder memories for next-token prediction. The prefiller is more expressive, with access to the full history, while the decoder operates with limited context.
The training process employs a memory consistency loss that aligns the decoder's memories with the prefiller's targets, allowing inference to use only the decoder (P) alone. This design reduces inference cost while maintaining high performance.
Empirically, Maglev improves validation loss and downstream pretraining benchmarks compared to sliding-window and latent recurrent transformer baselines. Additionally, sharing parameters between P and Q reduces parameter memory while preserving most of the performance gains, making it a practical solution for long-context applications.