In open-ended domains, large language models (LLMs) often lack verifiable rewards, making it difficult to formalize task preferences as effective supervision. Contexts can convey these preferences, but once distilled into the student, they provide little additional supervision. This motivates the use of contexts that evolve with student performance.
However, directly using evolving contexts as in-training supervision leads to unstable distillation targets and conflicting distributions. To address this, the researchers propose Flux-OPD, an on-policy distillation (OPD) paradigm that stabilizes the target and downweights conflicts.
The method is based on a decomposition of the reverse KL objective, revealing two key findings: the student is distilled toward the geometric mean of context-conditioned teachers, and the objective contains a conflict term that measures disagreements among these teachers.
Flux-OPD treats the differences between context-conditioned and context-free teachers as contextual difference signals, injects them as corrections into the context-free teacher anchor, and weights their strength using the conflict term as an indicator. Experiments on open-ended tasks show Flux-OPD outperforms existing OPD paradigms, highlighting the potential of combining teacher supervision with evolving contexts.