A newly posted Hugging Face paper examines why students trained through on-policy distillation (OPD) often produce responses that grow far longer than necessary, sometimes exhausting the entire generation budget. The authors trace a significant part of this behavior to a mismatch in termination tokens between base students and post-trained teachers.
Across the Qwen3, Llama, and Gemma model families, the researchers found that the two models can place their stopping probability on different EOS tokens even when their declared stopping sets are identical. In sampled-token OPD, this mismatch can suppress the student's preferred termination action without reliably transferring the teacher's preferred alternative.
The paper reports that simply aligning the decoding stopping set is not enough. Instead, aggregating probabilities over functionally equivalent EOS tokens and supervising them as a single semantic stopping action substantially mitigates mismatch-induced length inflation across all three model families. The authors illustrate the problem with a Qwen3 rollout in which the student reached the correct answer after 1,094 tokens but continued generating 7,098 redundant tokens.
The study also analyzes OPD across different K2-Horizon training stages, showing that termination preferences can shift considerably during training. Even after the proposed correction, a distinct late-stage length inflation reappears in the pretrained K2-Horizon student, indicating that termination mismatch is an important but not exhaustive explanation for OPD length dynamics.
The authors release an implementation that incorporates the proposed termination-handling corrections. Their experiments focus on single-turn mathematical reasoning, though they suggest the diagnostic may be useful more broadly. A practical tip offered in the discussion: inspect the teacher's EOS distribution at positions where the student terminates, and check whether the teacher assigns low probability to the student's stopping token while preferring a different, semantically equivalent EOS token.