Recent progress in world models has made visually realistic generated environments increasingly common, yet these environments remain largely silent. According to a new paper titled WorldSonus: Bringing Sound to Worlds, adding sound to world models raises three core challenges: real-time generation that keeps pace with interactive video streams, interactive control that responds to mid-stream sound instructions, and spatially aligned stereo that reflects scene geometry and camera motion.
To tackle these demands, the authors introduce WorldSonus, an interactive video-to-audio framework designed for real-time spatial sound synthesis in world models. For real-time generation, it uses a streaming causal autoregressive diffusion architecture that synthesizes audio chunks at a low real-time factor (RTF) of 0.41.
For interactive control, WorldSonus incorporates an audio-centric captioning pipeline with chunk-indexed prompt scheduling, which allows dynamic manipulation of sound events during generation. For spatial alignment, the framework leverages high-quality stereo supervision curated from diverse stereo and ambisonic data.
The paper reports that extensive experiments show WorldSonus, while tailored for world models, generalizes effectively to open-domain video-to-audio benchmarks, matching or outperforming state-of-the-art bidirectional models in both acoustic quality and spatial alignment. A project page is available at noizai.github.io/WorldSonus.