Researchers have introduced Learn2Play Bench, a new benchmark designed to evaluate how well LLM agents learn from experience in unfamiliar and dynamic environments. The work addresses a key limitation of existing benchmarks: many evaluate tasks whose rules are either provided in the instructions or already familiar to pretrained models, making it hard to separate genuine learning from interaction versus reasoning with existing knowledge.
To overcome this, Learn2Play Bench uses newly designed text-based games with rules that are novel or counterintuitive. Agents must acquire knowledge through interaction rather than relying solely on pretrained knowledge. The games provide reproducible feedback and automatic scoring, enabling controlled evaluation across repeated attempts. The benchmark also varies game instances to test whether agents can apply what they have learned to new situations.
The study evaluates how backbone models, self-evolving methods, and agent harnesses affect learning ability, yielding three main findings. First, on experience retention: retaining complete records of actions and feedback supports more effective learning than summarizing experiences into rules or strategies. Second, on the human-agent gap: top-performing human players achieve higher peak scores than the evaluated agents, exploring more varied strategies and repeating actions less. Third, on harness matters: with the backbone fixed, changing the harness can improve performance while reducing estimated inference cost.
Together, these findings offer insights into how LLM agents learn from experience and suggest directions for future work to improve their learning ability. A project website is available for further details.