Self-evolving large language model (LLM) agents rely on memory systems that learn from interactions. However, these systems face two intertwined challenges: trajectory-indexed utilities expand with history, dispersing limited feedback, and trajectory-level rewards are assigned jointly to co-retrieved memories, causing irrelevant experiences to receive misleading updates—a phenomenon known as the memory-reward trap.
To address these issues, researchers from Hugging Face introduce Reduced-Order Memory Reinforcement Learning (RoMeRL). RoMeRL represents the growing utility space using a fixed-dimensional per-task memory state, factorized by outcome polarity and memory dynamics. It incorporates new experiences through a fixed set of semantic coordinates whose contents are updated or replaced over time, concentrating feedback on a bounded utility support.
Theoretically, the authors show that this reduced-order parameterization increases the average feedback received by each utility coordinate and characterizes the steady-state occupancy of erroneous coordinates under a generic coordinate-transition model.
Empirically, across ALFWorld and LifelongAgentBench, RoMeRL improves task performance, reduces the Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces maintained memory size by 84.4%, and cuts LLM calls by 21.1%. These results demonstrate that reduced-order utility states support efficient self-evolving agent memory while limiting persistent reward contamination.
The code is available on GitHub at https://github.com/YOUNG-fnxm/RoMeRL.