Reinforcement learning (RL) with group-relative advantages has become the standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically combine rewards into a fixed weighted sum before standardization. This approach leads to two key issues: rollouts with different reward profiles can receive identical advantages, and all objectives are optimized with fixed weights regardless of their saturation level.
To address these problems, researchers from Hugging Face propose Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO). The method standardizes each reward objective independently and adaptively discounts its contribution based on a batch-level estimate of saturation. This dynamically reallocates optimization effort toward under-optimized objectives while maintaining performance on already satisfied ones.
The authors also demonstrate that saturation-aware reweighting can reverse the sign of an update, not just rescale its magnitude. In experiments, SA-MRPO improved the harder correctness objective over GDPO in 12 of 15 benchmark comparisons on mathematical reasoning, with gains up to 5% on AIME24. On adaptive reasoning, it improved accuracy on all five benchmarks by 3.8% on average and up to 9.2% on AMC23. On coding benchmarks, it improved pass rate by up to 2.3%, while keeping easier objectives near their satisfied levels.