A new research paper from Hugging Face, titled "The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation," provides a systematic analysis of how planning abilities emerge in foundation model agents. The authors introduce a unified, controlled multi-turn environment that allows precise manipulation of data and training conditions, enabling them to study planning across three key stages: acquisition during pre-training, shaping via post-training, and integration through multi-teacher distillation.
In the pre-training phase, the researchers found that explicit world model construction through chain-of-thought (CoT) state transition modeling leads to stronger long-horizon generalization. They note that atomic skills alone are insufficient for compositional generalization, but even a small amount of long-horizon data can help. Crucially, suboptimal trajectories severely impair performance because errors compound over long horizons.
For post-training, the paper examines two methods: GRPO (Group Relative Policy Optimization) and OPD (On-Policy Distillation). Using mutual information, the authors distinguish general planning patterns from task-specific planning knowledge. They identify three application regions for post-training: unnecessary, effective, and unsupported. OPD shows a broader effective region than GRPO under low-quality and long-horizon settings, as it provides more consistent update directions. However, distilling unseen procedures from a teacher with different knowledge may harm the student's prior world model without fully establishing new knowledge.
The third contribution is Multi-Teacher On-Policy Distillation (MOPD), which integrates capabilities by converging to a shared planning pattern across environments. Compatible patterns enable cross-environment generalization, partially shared patterns support continual learning, while completely conflicting patterns cause severe interference. The paper is accompanied by a project homepage, GitHub repository, and Hugging Face model and dataset links.