Constructing an interactive 3D open world from a simple user query is a complex task that requires understanding intent, planning scenes, and using 3D tools. However, existing methods are often evaluated on idealized queries, making it hard to compare how multimodal agents handle real-world requests. To address this, Hugging Face researchers propose VibeWorlding, a unified framework for benchmarking and training 'vibe worlding agents'—multimodal agents that autonomously infer user intent, plan layouts, invoke 3D tools, and reflect on feedback in a multi-turn interaction.
The framework includes VWE-BENCH, a benchmark with 2,616 high-quality 3D assets, 323 human-annotated seed worlds, and 6,828 reverse-synthesized queries, split into verified and unverified sets. It also introduces VibeWorlding-Gym, a joint multimodal RL post-training framework that integrates a sandbox environment (with asset retrieval, editing, and rendering as MCP tools) and a rubric-based verifier for physical feasibility and intent fulfillment.
Experiments reveal that even frontier models like GPT-5.5 and Qwen3.8-Max achieve below 60% success rate, with the bottleneck being precise 3D world editing. However, RL training eases this weakness: the open-source VibeWorlder-8B matches frontier models, while VibeWorlder-30B-A3B attains the best overall Pass@1 among all evaluated models.
This work provides the first agentic RL framework for end-to-end 3D world construction, offering a unified stack of assets, benchmarks, and training methods. It highlights the potential of open-source models to surpass closed-source systems when trained with reinforcement learning.