Hugging Face researchers have unveiled PlayWorld, a new benchmark designed to evaluate interactive video world models through the lens of autonomous agent players. The benchmark addresses a critical gap in comparing models that simulate future states based on user actions, where fixed action sequences often fail to produce comparable results across different systems.
The core innovation of PlayWorld is the introduction of a multimodal Agent Player that interacts with each world model in a closed loop, dynamically adapting its actions based on observed frames and action history. This approach mimics how a human player would pursue a long-horizon objective, such as turning 360 degrees to check environmental consistency or walking into water to observe ripple effects.
PlayWorld comprises 171 scenarios, each with a specified objective. Models are assessed along four core dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution, alongside basic ability metrics for video quality and controllability.
Experiments across nine state-of-the-art world models reveal that current systems remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution. The findings underscore the need for further development in this area. Code and data are available on GitHub.