Decoder-only language models typically entangle long-term memory and reasoning within a single parameter set, making it difficult to scale memory capacity independently. A new paper, Memory Decoder at Scale, tackles this by introducing a parametric long-term memory module that can be scaled separately from the base model.
The researchers scale their memory models up to 6.9B parameters and pretrain them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. To address this bottleneck, they develop a distributed pipeline for Faiss indexing and retrieval, along with sparse, batch-wise loading of kNN distributions.
Across model scales, the team finds that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. For instance, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34 on 17 benchmarks, surpassing Pythia-12B (37.24) with 39% fewer total parameters.
For Qwen3 Base models ranging from 0.6B to 14B, adding 1.7B domain memories improves the average score across three domains by more than 9 points at every scale. The results demonstrate that independently scaling pretrained memory offers a more parameter-efficient path to improving language model performance, supporting a "small backbone, large memory" paradigm.