Hugging Face researchers have introduced Puffin-World, a unified multimodal framework designed to jointly model physics, geometry, and appearance for physically consistent 3D world generation, reconstruction, and closed-loop exploration. The system aims to integrate physical understanding, spatial simulation, and 3D world generation without relying on external offline modules.
At its core, Puffin-World models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. This grounding in real-world camera properties ensures physically consistent and visually stable world generation.
The framework introduces a strategy for propagating physical dynamics across future frames, and it couples appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications, including mimic and self-calibrated world exploration.
To scale Puffin-World to complex scenarios, the team constructed Puffin-16M, a dataset comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various challenging motions. The code, models, and datasets have been released to foster further research in this area.