Reconstructing a 3D scene from a single image is most valuable, according to a new Hugging Face paper, when the result is an explicit scene program whose execution produces a scene that can be inspected, edited, and queried — rather than a fixed rendering or a static 3D output.
The paper introduces LEGO-Anything, an Image-to-Code framework in which a coding agent iteratively writes and executes Blender code, inspects the resulting scenes and renderings, and revises its program. To measure end-to-end scene recovery, the authors also present LEGO-Bench, a simulator-grounded benchmark built from 208 images spanning 104 diverse indoor and outdoor scenes. The benchmark scores artifact validity, visible-surface geometry, and rendered appearance separately, and its simulator-grounded design is intended to support extensibility and precise automatic evaluation.
Among the evaluated agents, GPT-6-astra achieved the strongest overall results, with 53.4% indoor and 39.6% outdoor scores. The authors note that substantial gaps remain between producing valid scene artifacts and faithfully recovering scene geometry and appearance.
Analysis of agent construction trajectories surfaced three recurring problems: weak scene initialization, regressive edits during iteration, and unreliable self-evaluation. These findings motivated LEGO-Plugin, a training-free harness plugin for more controlled iterative scene construction, which improved all six evaluated models, with relative gains of up to 62.7% in overall score.
Finally, the paper tests whether reconstructed scenes can represent natural images and support vision tasks. In LEGO-World, object detections, instance masks, and relative depth are derived as deterministic queries on scenes reconstructed by GPT-6-astra. These readouts show non-trivial performance across all three tasks but fall well short of specialized vision models, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.