Humans can localize a sound source and combine that information with visual cues almost effortlessly, but embodied agents still struggle with the same task. According to a new research paper, it has remained unclear how to properly evaluate and model spatial audio understanding in embodied settings.
To close that gap, the authors introduce OmniEchoBench, described as a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. The benchmark spans six tasks built on 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples. The audio was collected as first-order ambisonics (FOA) from 30 real-world environments.
The team also developed a controllable rendering pipeline for spatial audio that maintains geometric consistency among sound sources, visual observations, and agent trajectories, enabling scalable training supervision.
Building on this foundation, the researchers propose OmniEcho, a spatially aware omni-modal model that pairs an FOA spatial encoder with a pretrained semantic audio pathway.
Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. In sound-guided navigation, it reaches a level close to that of traditional vision-language navigation. The authors conclude that spatial audio can be a valuable signal for embodied scene reasoning and navigation, while identifying fine-grained spatial localization and distance estimation as important open challenges.