Researchers at Hugging Face have proposed a novel post-training technique called RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) to enhance language model reasoning. The method addresses a key limitation in on-policy distillation (OPD), where dense per-token supervision is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity.
RISE constructs a synthetic teacher directly from the model's own reinforcement learning with verifiable rewards (RLVR) trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor—in parameter or output logit space—RISE converts a sparse outcome-induced parameter update into a dense token-level target, eliminating the need for external models or privileged conditioning.
The method combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step.
Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms both RLVR-only training and on-policy self-distillation across all settings, suggesting a promising path for scalable, self-improving AI systems.