Hugging Face has released a research paper detailing Matrix-Game 3.5, an advanced interactive world model that pushes the boundaries of real-time simulation. Building on its predecessor Matrix-Game 3.0, the new model targets persistent, interactive virtual environments for applications in gaming, robotics, embodied AI, and extended reality (XR).
The core challenge addressed is maintaining stability over long horizons while preserving scene geometry, dynamic consistency, and camera control during autoregressive generation. Matrix-Game 3.5 introduces three key improvements to tackle this.
First, a unified geometry-aware memory framework combines explicit 3D patch retrieval with projective camera conditioning, using patch-memory and tiled-PRoPE components that add no learnable parameters. This enables geometry-consistent camera control and faithful recall of scenes over extended periods.
Second, a static-dynamic disentangled world representation separately models static scene geometry and dynamic subjects, ensuring geometric consistency and subject identity are maintained throughout long-horizon generation.
Third, a two-stage progressive real-time distillation framework converts a bidirectional diffusion model into a few-step causal generator via Perceptual Flow Matching and curriculum-based Self-Rollout DMD. This allows for minute-long real-time interactive generation.
Experiments across a unified training corpus—including Unreal simulation environments, open-world games, and internet videos—demonstrate strong performance in long-horizon scene recall, precise camera control, subject consistency, prompt-driven world generation, and stable real-time open-world interaction.