Efficient scaling of large language models has driven interest in sparse capacity mechanisms such as Mixture-of-Experts and, more recently, conditional memory — token-indexed embedding tables that give a model cheap parametric lookups alongside its main backbone.
A new paper posted on Hugging Face, titled MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup, argues that existing memory-embedding methods share a key limitation: they retrieve memory through a deterministic function of a token's surface form. That design collapses different contextual senses of the same token — the paper's example is python as a programming language versus the animal — into a single fixed entry.
MoME addresses this by replacing each token's single memory row with a mixture of M slots, and using a learned gate over the hidden state to decide which slots to read at each position. The result is a context-aware memory mechanism rather than a fixed lookup.
In controlled pretraining experiments spanning nanochat, Llama-3/MobileLLM and Qwen3 backbones, the authors report that MoME improves over Value Embedding, Bigram and STEM baselines under iso-parameter and iso-training-FLOP settings. They also describe a more promising memory-size scaling trend at sub-billion scale, while keeping training and inference efficient.
Qualitative routing analyses on polysemous tokens suggest the learned mixture carries some degree of semantic interpretability, dispatching the same surface token to distinct memory slots under different senses. The authors additionally train a sub-billion-parameter model that they say reaches competitive CORE-22 performance against similarly sized base models, including Qwen3-0.6B and Llama 3.2-1B. Code and pretrained models are said to be publicly available.