Reinforcement learning (RL) has become a key driver of progress in large language models, but its substantial memory requirements have limited wider adoption. A new research paper introduces LoGRA, an approach to RL post-training that reduces memory by retaining useful learning signals in low-rank gradient sketches.
These compact representations serve two purposes: they support model updates and enable efficient policy synchronization. To prevent overly large updates from destabilizing learning, the authors pair gradient compression with predicted-KL step control, which estimates policy changes before each update and adjusts the update's magnitude accordingly.
Across reasoning tasks, LoGRA reduces average training memory by up to 45.7% without sacrificing performance, according to the paper. The method also enables stable training of a 27B-parameter model for more than 1,100 steps on a single eight-GPU node — a scenario where dense Adam runs out of memory.
The authors say this makes previously memory-infeasible RL training practical. Code is available in the Molt library.