Hugging Face has introduced Luce, a novel 3D representation designed for high-fidelity image-to-3D generation. The model unifies geometry and physically based rendering (PBR) materials within a voxelized multimodal Gaussian cloud, using dedicated Gaussian primitives for each modality such as albedo, metallic-roughness, and surface normals.
Luce employs a variational autoencoder to compress this representation into a unified material-aware latent space. A rectified-flow transformer then generates this latent from a single image, conditioned on multi-layer features from a pretrained image encoder that preserve both semantic context and fine spatial detail. The latent decodes into relightable PBR Gaussians and an optional textured mesh with a tangent-space normal map.
On the Toys4K benchmark, Luce achieves state-of-the-art single-image-to-3D generation, improving FID by 28% over the strongest baseline. The team also introduced a new benchmark of AI-generated images, where Luce improves the CLIP image-alignment score over the best baseline (0.8519 vs. 0.8299).
According to the paper, Luce generates relightable, geometrically accurate, and materially faithful assets that preserve fine details such as text, logos, and inscriptions, making it suitable for integration into standard rendering pipelines.