Language-model agents are typically trained with reinforcement learning that supplies a single reward per episode. Privileged self-distillation aims to enrich that signal by letting the same policy, when given a skill, teach its skill-free counterpart through token probabilities.
A new paper posted to Hugging Face, PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning, questions whether token probabilities are the right channel for that transfer. The authors identify two phenomena. The first, Invisible Advantage, notes that adding a skill in context raises WebShop success from 42.2% to 56.2% while changing the probabilities of fewer than a quarter of the sampled tokens. The second, Much to Align, finds that a skill alters the hidden states of more than 80% of response tokens, in a way that linear probes can trace back to the specific skill.
To exploit this, the authors propose Privileged Representation On-policy Self-Distillation (PR-OPD). After a GRPO warm start, the policy writes a hindsight skill for each trajectory, re-reads its own responses with that skill acting as a stop-gradient teacher, and aligns its projected hidden states to the teacher's at every layer alongside the reward objective. The method requires no external skill library, no separate teacher model, and adds no inference overhead.
On ALFWorld and WebShop with two backbones, PR-OPD reports the best overall results in every setting, improving over GRPO by up to 4.7 points in ALFWorld success and 14.0 points in WebShop accuracy. Code is available at github.com/balibata/PR-OPD.