On-policy self-distillation has emerged as a promising post-training technique for large language models (LLMs), but most existing approaches depend on external signals such as ground-truth labels, environment feedback, or guidance from larger models. This reliance limits their applicability and moves away from the idea of true self-improvement.
In a new paper, researchers propose Unsupervised On-Policy Self-Distillation (U-OPSD), a method that requires no external supervision at all. Instead, U-OPSD leverages the model's own generations: it samples multiple rollouts, then constructs a pseudo-solution by majority vote under a self-consistency threshold. The model then conditions its distribution on this pseudo-solution and distills itself on the completions where it disagrees, effectively correcting itself precisely where it is confidently wrong.
The method was evaluated across diverse benchmarks, base models, and training settings. On five mathematical reasoning benchmarks (AIME24, AIME25, HMMT25, MATH500, and AMC23), U-OPSD improved over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, respectively. It also outperformed supervised OPSD by 3.2% and 2.3% on average in those settings.
In thinking mode, U-OPSD matched OPSD performance, ahead by 0.9% at 4B and level at 8B, while surpassing GRPO by 0.7% and 1.1%, respectively. These results suggest that unsupervised self-distillation can be a viable alternative to supervised methods, potentially reducing the need for labeled data or external feedback in LLM post-training.
The code is available on GitHub, allowing researchers to reproduce and build upon the findings.