Researchers at Hugging Face have introduced Marionette, a new world model designed for interactive games with articulated characters. Unlike traditional models that autoregressively predict visual observations in pixel or latent space, Marionette explicitly models the evolving world state, delegates exact geometric computation to a fixed, zero-parameter renderer, and leaves the neural model to synthesize appearance.
The architecture consists of three main components. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state, including multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls.
The experiments demonstrate two key properties. First, the predicted world state is directly controllable: forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behavior is determined in the state and can be repaired there. Left free, two generated characters drift to 21.2 meters apart (recorded sessions stay near 5 meters) and a third of frames show ground penetration. Two rules imposed on the explicit state—a terrain collider and a separation cap—cut penetration by 66% and keep the pair engaged, with no change to the observation model.
The authors note that routing appearance through the predicted state costs no detectable fidelity, with an FVD of 831 against 799 for recorded pose. This approach offers a promising direction for improving consistency and controllability in interactive world models.