On-policy distillation (OPD) trains a student model to match a teacher's next-token distributions along the student's own trajectories, and has delivered substantial empirical gains. Generalized variants go further by extrapolating an implicit reward in output space, allowing the student to surpass the teacher. However, the language-model head attenuates this change anisotropically: much of the change encoded in the teacher's hidden states reaches the logits at only a small fraction of its weight, and the sampled-token log-probability ratios that output-space extrapolation relies on inject noise that the extrapolation then amplifies, making training unstable.
Researchers observe that reinforcement learning (RL) shifts a model's internal representations relative to its base checkpoint, and that the direction of this shift can be measured at every layer. Building on this, they propose RIDE (RL-Induced Direction Extrapolation), which extrapolates the RL-induced change directly in representation space. At every layer and token position, RIDE computes the residual between the teacher and its pre-RL checkpoint and regresses the student's hidden states toward targets displaced beyond the teacher along this residual.
Conditioned on a sampled trajectory, this regression is equivalent to maximizing a linear directional reward defined by the residual under a quadratic penalty centered at the teacher. This makes explicit how the objective moves the student along the RL-induced direction while limiting its deviation from the teacher.
Across four base/RL-teacher pairs spanning different scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean does so. It consistently outperforms output-space extrapolation, which degrades the student whenever the teacher is close to its base. The paper includes an optimization interpretation, mechanistic analyses, and ablations. Training code and checkpoints are planned for release upon publication.