Understanding 3D scenes is crucial for embodied intelligence, requiring joint reasoning over visual and geometric cues. However, existing Multimodal Large Language Models (MLLMs) often rely on fixed modality combinations, which can introduce semantic noise from irrelevant modalities and underutilize informative ones, leading to wasted computation and diluted reasoning.
To address this, researchers from Hugging Face propose SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. SmartMage incorporates two key modules: a Semantic-guided Modality Adaptive RouTng (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and a Modality-Aware Gating Expert (MAGE) module that uses modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning.
Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks and attains competitive results on RGB-only video understanding benchmarks. In the diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness.
For more details, visit the project page: https://yuecheong.github.io/SmartMage/.