In a new research paper, Hugging Face scientists investigate the role of training data in on-policy distillation (OPD), a technique that combines student-generated rollouts with dense token-level supervision from a teacher. While previous work has focused on algorithmic aspects, this study explores the data-minimal limit by training on a single query.
The authors find that one-shot OPD continues to improve for hundreds of steps and recovers most of the performance gain of full-data OPD across various task domains and model families. They explain this through state coverage—the fraction of states visited by full-data OPD that a query set's rollouts reach. A single query already covers 71.5% of these states, mostly within the first 100 steps.
Adding semantically distinct queries increases both coverage and validation accuracy, with 16 queries reaching 98.9% coverage and matching full-data training. However, alignment slows at a similar pace regardless of whether OPD trains on one query or the entire dataset, suggesting that the bottleneck is not data availability but the algorithm's step efficiency.
The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. Even content-light templates and off-domain queries approach the real-query baseline, indicating that task content and induced state coverage can be decoupled. The authors hope these findings steer future research toward improving the step efficiency of OPD.