Long-horizon agents must compress their interaction history to keep operating inside a finite context window, but a text-only summary may not carry enough information for every later decision. A new paper posted on Hugging Face, REMORY: Learning Residual Memory for Context Compaction, proposes supplementing that summary with learned memory representations rather than relying on words alone.
REMORY is described as a neural memory network that generates a bounded sequence of soft memory tokens. Conditioned on the summary and appended after it, these tokens act as an analogue of a residual connection along the sequence dimension. The network is trained so that a frozen LLM, given the summary plus the tokens, approximates the continuation it would have produced with the full history.
On SummHay, the paper reports that REMORY improves source attribution at nearly unchanged insight coverage, and approaches the full-context joint score while using only 5.2% of the input positions. Across long-horizon agent benchmarks, the authors say Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory, along with substantially fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.
A discussion thread raised a practical question: whether the soft tokens preserve any human-traceable link back to the original step when a summary loses provenance. The authors acknowledge that the tokens are not directly human-readable and that their information is recovered through the LLM — which is also the training objective, since REMORY learns representations the frozen model can interpret and use.
One hypothesis offered in the thread is that the soft tokens may function as a form of associative memory, connecting the model's working context with external long-term memory such as saved files or memory.jsonl, potentially preserving cues about where relevant information resides so it can be retrieved again.