Reinforcement learning (RL) for large language models (LLMs) often relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion (whether the policy has moved too far from the behavior policy) and a direction criterion (whether the update pushes it farther away). Recent work, DPPO, improved the proximity criterion by using a probability divergence instead of a ratio-based test, but retained PPO's direction criterion.
The authors observe that the ratio-based direction criterion is a single-sample proxy that can disagree in sign with the change of the divergence defining the proximity criterion. To address this, they propose the predictive divergence mask, which asks whether the next policy-gradient step will increase or decrease the same divergence used by the trust region. For discrete softmax policies, they derive this prediction in closed form.
Because production rollout engines expose only a truncated (top-K) view of the vocabulary, the authors develop two lightweight top-K estimators for this prediction. Detailed analysis shows the divergence-based direction is better aligned with the realized change of the divergence than the sampled ratio, and the resulting masks improve RL training across model scales and precision settings.