Group Relative Policy Optimization (GRPO) is a widely adopted method for training reasoning language models. It computes advantages by centering and normalizing rewards across rollouts of the same prompt. When multiple rewards are involved, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation.
According to the paper, the corresponding variance equals the sum of all pairwise reward covariances. For a fixed centered reward, a larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, the authors note that correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards.
To address this, the researchers introduce Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components.
The team compared CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results reportedly show improvements across all three domains.
The code is available at https://github.com/HKUST-KnowComp/CorrGRPO.