Multi-reward reinforcement learning aims to train large language models to satisfy several behavioral objectives at once. A common approach, reward-wise normalization as used in GDPO, preserves reward-specific relative information within rollout groups, yet some objectives still learn far more slowly than others.
In a new paper, researchers trace this imbalance to advantage energy — the sum of a reward's squared advantages over a batch. Under idealized GDPO normalization, they show this energy is proportional to active-group density: the fraction of rollout groups in which a reward yields nonzero relative advantages. Sparse rewards therefore end up drowned out at the batch level.
To address this, the authors propose Density-Aware Reward Aggregation (DARA). It applies an inverse-square-root density correction that gives more weight to signals from rewards that are active less often, with the theoretical aim of equalizing energy across rewards. Crucially, DARA computes its weights from each rollout batch, so it adapts as reward activity shifts during training without altering the underlying policy optimization objective or requiring tuning.
Experiments on tool calling and mathematical reasoning show DARA learns the targeted behaviors faster than GDPO. It reaches high format compliance in up to 26% fewer training steps on tool calling, and near-saturated length compliance in up to 65% fewer steps on mathematical reasoning, while remaining competitive in final performance.
The code is available at github.com/zhaihaotian/DARA.