Multi-turn agents trained with reinforcement learning typically receive only a single scalar reward per trajectory, a sparse signal that has motivated self on-policy distillation (OPD): a self-teacher with privileged task skills supplies dense token-level supervision so that a skill-free student can internalize those skills.
According to the paper RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning, this recipe breaks down in agentic tasks for two reasons: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision depends on the training stage.
The proposed method, RetireOPD, first optimizes a decoupled, skill-conditioned teacher using environment rewards, then trains a skill-free student jointly with RL and OPD. Instead of following a predefined distillation schedule, it uses Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training continues with RL alone.
Across Qwen2.5 models ranging from 1.5B to 7B parameters, RetireOPD is reported to improve ALFWorld success rate over an RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%. The authors also state that the student surpasses its own skill-conditioned teacher in every setting.
Code for the work is available at github.com/ZJU-REAL/SDAR.