Reinforcement learning with verifiable rewards (RLVR) is a key driver for improving long chain-of-thought (CoT) reasoning in large language models. However, critic-free methods like GRPO uniformly distribute response-level advantages across all tokens, ignoring their unequal contributions to the final outcome. On-policy self-distillation (OPSD) offers dense supervision but implicitly assumes that likelihood shifts encode reliable answer-aligned information.
Researchers at Hugging Face tested this assumption by re-scoring trajectories under opposing outcome conditions. They found that most affected tokens shift in the same direction regardless of correctness, with few sign reversals and substantial overlap in optimization signals. Large shifts concentrate on substitutable surface-form tokens, while reasoning-critical tokens are less sensitive.
Based on these findings, they propose Counterfactual Sensitivity Credit Reallocation (CSCR), a simple extension of GRPO that reduces credit for highly sensitive tokens and renormalizes token-level advantages to preserve the original credit budget and verifier-determined direction. On long-CoT mathematical reasoning benchmarks, CSCR consistently outperforms GRPO with the same number of policy updates.
Targeted ablations confirm that privilege-induced directions are unreliable, moderate downweighting is most effective, and stronger modulation destabilizes optimization. The method also improves over self-distillation baselines across multiple model scales.