Researchers have introduced ReSPO (Reshaped Sequence Policy Optimization), a new policy-optimization method aimed at a persistent problem in reinforcement learning from verifiable rewards (RLVR): reusing rollouts across multiple policy updates. This reuse widens the mismatch between the current policy and the policy that generated the data, a setting known as off-policy learning.
The paper identifies a sign-dependent gradient starvation issue in clipped policy optimization. Clipping suppresses under-generated positive responses that fall in the low-importance-weight tail, while allowing severely over-generated negative responses to dominate the high-weight tail. In effect, useful learning signals are lost while unstable ones are amplified.
ReSPO addresses this by replacing clipping with a smooth, two-branch sequence-level kernel derived from an α-divergence variational objective and an exponential variance-control tilt. The positive branch keeps a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses.
According to the authors, ReSPO can learn effectively from long positive reasoning trajectories early in training, even when accumulated policy drift pushes them into the low-importance-weight tail. On dense and MoE Qwen3 models, the method accelerated early optimization, improved final training scores, and achieved higher held-out benchmark performance under rollout reuse.
The authors frame the work as a validation of importance-weight tail control in off-policy learning, and note that gains grow as the number of sampled responses increases. Code is available at github.com/yhangchen/ReSPO-code.