Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern models handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding, according to a new research paper titled Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering.
Prior efforts have tried to close this gap by injecting 3D awareness into MLLMs, either by improving fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models. Yet the paper notes that a substantial gap to human reasoning persists.
The authors revisit how humans approach spatial reasoning: rather than relying on fine-grained geometry cues, people roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, Imagine3D-LLM learns to assemble a similar compact 3D representation and conditions its answer on that representation.
Concretely, the model appends a small set of learnable summary tokens after the image tokens, decodes them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and trains jointly with the standard next-token prediction objective.
Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model.
As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.