Reinforcement learning with verifiable rewards (RLVR) has become a key method for post-training large language models (LLMs), but it often relies on coarse outcome rewards that give limited guidance on intermediate reasoning steps. Existing approaches like process reward modeling and on-policy distillation have constraints, such as needing specialized reward models or assuming identical reasoning patterns between teacher and student.
In a new paper, researchers introduce Cliff, a reward shaping strategy that leverages an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. The rollout is then split into a correct prefix and an incorrect suffix, with token-level advantages assigned accordingly: positive for the correct part and negative for the rest.
The method is simple and general, requiring no additional training of a reward model. Experiments across 12 different scenarios show that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even when the teacher model is of modest capability.
The authors also analyze the role of "ground truth" in Cliff and investigate its training dynamics, establishing it as an effective approach for enriching RLVR with finer-grained supervision.