Hugging Face has released a research paper introducing DreamX-Creator 1.0, a compact 7B-parameter model designed for native joint audio-video generation at 2K resolution. Unlike many existing systems that either omit audio or synthesize it separately, DreamX-Creator generates both modalities simultaneously, enabling reciprocal modeling of visual dynamics and acoustic events.
The system conditions on a first frame and a text prompt, jointly denoising modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, where token- and head-wise output gates modulate each active cross-modal attention-head output.
Training involves a unified Audio-Video Data System that constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training includes two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning post-trains the generator with Modality-Aware Multimodal Feedback, routing video-, audio-, and cross-modal feedback to the corresponding streams.
For high-resolution output, the Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. The authors report performance competitive with state-of-the-art open-source systems.
By releasing the compact 7B generator and 2K Refiner, the team aims to democratize native audio-video generation and provide an accessible foundation for future research. The code is available on GitHub.