Researchers at Hugging Face have unveiled WorldSculpt, a novel approach for generating compositional 3D representations of cluttered scenes containing hundreds of objects. The method tackles a key challenge in computer vision: reconstructing individual object meshes from multi-view observations when objects heavily occlude one another, a scenario that stumps traditional geometry-based techniques.
Existing geometry-based methods typically reconstruct the entire scene as a single mesh, leaving incomplete geometry in occluded regions. While compositional methods with generative priors exist, they are largely limited to simple scenes. WorldSculpt overcomes this by adapting a strong single-object 3D generative prior, specifically Pixal3D, to multi-view observations. The model is extended with a multi-view conditioning pathway that grounds object generation in multiple posed observations.
Remarkably, although the model is finetuned entirely on single objects in canonical space, it generalizes to large, densely cluttered scenes without any scene-level training. This demonstrates the feasibility and scalability of the paradigm, as the model can handle severe occlusion and hundreds of objects.
To evaluate their method, the team introduces UE-MeshyScene, a photorealistic benchmark of densely cluttered scenes with per-object annotations and ground-truth meshes. Across single-object, controlled multi-object, and UE-MeshyScene evaluations, WorldSculpt consistently outperforms prior approaches, with larger gains as scene complexity and occlusion increase. The method also shows broader applicability by converting generated 3DGS worlds, such as Marble and HY-World 2.0, into compositional mesh scenes.