On-policy distillation (OPD) for LLM agents typically requires fresh student rollouts and teacher queries at every update, making it prohibitively expensive for multi-turn tasks. Researchers from Hugging Face introduce ReOPD (Replayed-Prefix On-Policy Distillation), an off-environment alternative that leverages pre-collected teacher trajectories as replayed prefixes to provide dense per-step supervision without new environment interactions.
The key challenge in multi-turn OPD is the prefix trap: making histories more student-on-policy improves relevance but can query the teacher on histories where its target is unreliable, creating a two-sided distribution shift. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design, implemented via a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes.
Experiments across mathematical reasoning (Python) and search environments, with multiple teacher and student model scales, show that ReOPD preserves or improves OPD-level accuracy while using zero tool calls during student training. It is at least 4× faster per rollout than standard OPD, turning expensive agent-environment interaction into a reusable offline resource.
The method enables scalable distillation across tools, tasks, and environments, making it a practical solution for training smaller models from larger ones without costly live rollouts.