Large language model (LLM) agents have shown promise in complex interactive tasks, but their reinforcement learning (RL) is often hampered by sparse rewards—a long multi-turn trajectory may yield only a single outcome-level signal. To address this, researchers from Hugging Face propose Persistent Consistency Self-Distillation (PCSD), a novel training objective that combines dense teacher guidance with sparse environmental feedback.
Traditional on-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods either rely on isolated token-level discrepancies, which are sensitive to noise, or assign a shared step-level weight that overlooks positional variation. PCSD instead derives token-level distillation weights from the local persistence of teacher-favoring signals.
PCSD employs adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights via sigmoid gating. The resulting objective is jointly optimized with GRPO, blending dense teacher guidance with sparse environmental rewards.
In experiments without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points. It remains competitive on WebShop and gains 15.8 points over GRPO on an unseen ALFWorld split.