Researchers from Hugging Face have published a paper on Experience Distillation, a method that enables agents to learn from their own interaction histories without requiring additional environment interactions. The work addresses a key limitation of in-context learning: while it is highly sample-efficient, its benefits vanish once the experience is removed from the context window.
The team proposes distilling the agent's collected experience directly into model weights, a process they term Experience Distillation. Unlike context distillation, which typically requires further environment interaction, their implementation operates solely on already-collected data. Experiments were conducted on 749 curated software-engineering tasks and six text-adventure games.
Results show that Experience Distillation retains at least 64.8% of the performance gains from in-context learning across both domains. In contrast, direct supervised fine-tuning on the same experience recovers only 3.8% of those gains. Compared to classical reinforcement-learning baselines, in-context learning followed by Experience Distillation matches performance while requiring at least 9.6× fewer environment samples.
The paper highlights that the approach is particularly valuable when environment interactions are costly, such as running time-consuming experiments or obtaining human feedback. By internalizing experience into weights, agents can maintain learned behaviors without carrying large context windows.