Researchers behind a new paper titled AgentGarten: Code Worlds for Evolving Agents propose a framework for building real-time interactive environments that are both faithful and realistic. According to the authors, what an agent can learn is bounded by the environment it practices in, and achieving consistent state, rules, and dynamics alongside real-world-like visuals across diverse worlds remains a bottleneck.
AgentGarten addresses this by coupling simulators and game engines with a shared neural renderer. Its simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer generates visual observations from structured conditions exported through a common interface.
To build the renderer, the team adapts a pretrained video model to geometry conditions, distills it with a technique they call Adversarial Forcing, and optimizes inference for real-time interaction. Adversarial Forcing makes history prefilling differentiable through exact replay, so losses on later predictions update how the renderer encodes prior observations, and it adds real-data adversarial supervision to improve visual quality.
Inside these worlds, agents perceive the environment through visual observations, interact with it in real time, and improve by distilling each round of experience into playbooks that subsequent agents inherit and refine. The authors report a substantial gain in learning efficiency: agents learn from just four rounds, compared with millions for a conventional reinforcement learning counterpart.
In a revisited hide-and-seek setting, hiders build shelters by round four and seekers use ramps by round ten. Because new worlds can be written as code and rendered through the same interface, the authors argue environments can scale in number and difficulty alongside their agents — a step toward agents that keep evolving through interactive experience.