Iterative self-distillation allows large language model (LLM) agents to learn from successive deployments, offering a potential route toward recursive self-improvement. However, experiments with existing methods reveal a collapse in deployment performance across cycles, alongside a decline in task performance when privileged information (PI) is available. This matters because the student model of one cycle becomes the teacher of the next.
To address this, researchers propose ReSAIL (Retentive and Selective Augmentation for Iterative Self-Distillation), a plug-in augmentation for iterative PI-based self-distillation. It has two complementary components: trajectory-balanced selective distillation, which prioritizes interaction steps where PI most strongly changes the teacher's predictions and balances the resulting distillation losses across trajectories; and privileged retention, which regularizes the student's PI-conditioned output distributions toward those of the frozen teacher at both selected and unselected steps, preserving PI-conditioned behavior for supervision in the next cycle.
Evaluated with Qwen3-4B and Qwen3-8B over three deployment cycles on ALFWorld and TextCraft, ReSAIL sustained substantial gains across model scales. At cycle 3, it improved success rates by an average of 22.5 percentage points over the corresponding SDPO/OEL baselines, across two model scales and three evaluation settings (ALFWorld ID, ALFWorld OOD, and TextCraft).
The authors also report that sensitivity-guided selection of offline data improves action prediction accuracy for multimodal GUI agents on the AITZ benchmark. They describe these findings as the first evidence that a more robust learning mechanism can effectively mitigate performance collapse in iterative agent self-distillation over deployment trajectories.
Code, training and evaluation scripts, and model checkpoints are publicly available.