Distilling a capable language-model agent into a smaller model usually means aligning the student's behavior with the teacher's. But that raises a basic design question: at which states, and at what granularity, should this alignment happen?
According to the paper, offline imitation offers dense supervision but is token-specific and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses that mismatch, yet it typically requires teacher token probabilities and compatible tokenization.
The authors argue that semantic actions are a more natural unit of supervision for agent distillation. Their method, ActionOPD, is an action-level on-policy distillation approach motivated by maximizing expected teacher-action likelihood at states the student actually encounters.
At each such state, ActionOPD samples actions from both the student and a black-box teacher, then rewards student actions based on their semantic agreement with the sampled teacher actions. Through policy optimization, the student's action-level behavior is aligned with the teacher's decisions — without reproducing teacher-written reasoning or accessing teacher token probabilities.
The authors report that training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3–6.9 percentage points across four out-of-distribution benchmarks. ActionOPD is said to scale favorably with student size and teacher capability, and it does not require the teacher and student to share a model family or tokenizer. On τ^2-bench, it is reported as competitive with offline distillation and token-level OPD. The paper also explores Self-ActionOPD using a same-model teacher with privileged context.