Large language models (LLMs) are increasingly deployed in interactive environments where they accumulate behavioral experience. Yet most existing agent benchmarks treat them as fixed policies, leaving a critical question unanswered: can an agent actively test its behavior, judge the resulting experience, and use that to improve future decisions?
To address this, researchers at Hugging Face introduce S3Gym, an interactive benchmark that evaluates LLM self-improvement through three coupled capabilities: Self-Testing, Self-Judging, and Self-Improvement. The benchmark separates permissive exploration from strict held-out evaluation and is instantiated across seven text-based games with executable environment verifiers.
The study compares three pathways for incorporating interaction experience: direct history in-context learning (ICL), score-conditioned summary memory, and parameter training. Results reveal that self-improvement is neither automatic nor uniform. Context-level experience helps some model–game pairs, but the most effective pathway depends heavily on task structure. Summaries are beneficial when experience can be compressed into reusable strategic rules, yet often underperform raw history when precise, state-contingent information is required.
Parameter training yields substantial gains on some tasks but also shows unstable improvement and severe negative transfer on others. The authors conclude that recognizing successful actions is insufficient; agents must transform feedback into executable and transferable policies. S3Gym provides a unified framework for diagnosing bottlenecks that prevent agents from converting interaction experience into reliable self-improvement.