Researchers have identified a previously overlooked failure mode in the critics used by Proximal Policy Optimization (PPO) when training large language models. In a paper titled Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening, the authors describe how state values estimated from multiple Monte Carlo continuations can change sharply across intermediate states, while the critic's predictions remain comparatively flat — a phenomenon they call Value Flattening.
The team observed the same behavior in a controlled FrozenLake environment and found that it becomes more pronounced as the state space grows. Their theoretical and empirical analyses link Value Flattening to an implicit variance penalty in the critic loss and to redundant updates caused by temporally correlated states that produce similar gradients.
To address the problem, the authors propose SParse Proximal Policy Optimization (SP3O), which applies the value loss to only a few well-separated states within each response, targeting both underlying causes simultaneously.
Experiments on Qwen3-Base show that SP3O, with just three states supervised per response, can mitigate Value Flattening and consistently improve the learned policy across model sizes and evaluation suites.
The authors conclude that Value Flattening is an important yet overlooked failure mode of critic learning in standard PPO, and that a simple sparse supervision strategy offers an effective remedy. Code and project details are available via the paper's linked repository.