Language models often need to internalize information that arrives over time and keep it through many subsequent updates. A new paper posted on Hugging Face examines this challenge under the name long-horizon memorization: a model learns 100 query-answer tasks through continual supervised fine-tuning, without retaining earlier training examples and without receiving task identifiers at inference.
According to the authors, sequential updates cause catastrophic forgetting, and no single continual learning mechanism they evaluated maintains strong retention at this horizon. Their hypothesis is that mechanisms addressing complementary sources of forgetting will work better when composed.
The paper organizes these compositions along two design dimensions. Data, function, and weight anchors specify what prior information each update should preserve, while low-rank allocation rules determine where successive updates are retained. To test the idea systematically, the team built three distinct 100-task memorization datasets, introduced task-level successive halving to search the combinatorial design space, and used a factorial experiment to measure individual and interaction effects.
The best method combines all three anchors with merged LoRA. It ranks among the top 3 methods on all datasets and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9% — a 28-fold improvement. The data anchor and merged LoRA provide the largest average gains and interact super-additively on all three datasets.
The authors conclude that composing complementary mechanisms substantially improves long-horizon memorization beyond what any individual mechanism achieves.