Researchers have introduced SepGen, a model that extends a pretrained audio-video generator to output not only a video and its mixed soundtrack but also one individual waveform per captioned source — a "stem" — within a single joint sampling run. The work is described in a paper shared on Hugging Face.
The motivation is spatial audio rendering. A 4D audio-visual scene includes a video, the dynamic geometry it depicts, and multiple sound sources, each with its own position and trajectory. Rendering such a scene from a new viewpoint requires every source to be available as a separate waveform so it can be localized and propagated to the observer before the signals are mixed. Existing joint audio-video generators emit only a single audio mix, leaving sources inaccessible individually.
SepGen supports two complementary modes. In generation mode, each source caption specifies what its stem contains; a two-speaker dialogue, for example, emerges as one stem per speaker in the original turn order. In separation mode, the input audio mix remains clean while captions specify what to extract, allowing the model to decompose a recording from a free-text description.
The authors evaluate generation on scenes synthesized from text, and separation on scenes rendered by other generators as well as on real recordings of speech, music, and sound effects. According to the paper, given an audio mix and captions carrying the spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and retains its lead even when the spoken lines are removed from the captions.
Code, checkpoints, and datasets are available, and the project page includes a demonstration of walking into a scene with spatial sound.