Hugging Face has released a research paper introducing SwanTale, a unified model for multi-speaker speech and audio generation. The model is designed to handle both zero-shot and instruction-based tasks, catering to creators in animation dubbing, audio drama, movies, advertising, games, podcasts, and short-video production.
In these scenarios, creators often need to design voices without reference recordings, control speaker styles via natural language, and incorporate acoustic scenes with environments and audio effects. SwanTale addresses these needs by supporting both instruct tasks (using a caption of environment, speaker styles, and fine-grained content) and zero-shot tasks (using reference audio alongside the same content).
The work introduces SwanData-Caption, a data pipeline that cleans raw speech and audio data, adds targeted synthetic coverage, and annotates diverse multi-level captions. On the model side, SwanTale incorporates SwanVAE for high-quality multi-audio-modality generation, along with reward-conditioned quality control, Engram conditioning, and a Unified Mixture-of-Experts (MoE) architecture for multi-task modeling.
Training employs curriculum learning and GRPO post-training to progressively strengthen capabilities. Experimental results show SwanTale leads on multiple key zero-shot and instruct metrics, achieving the best expressiveness scores in both tasks and supporting complex instruct generation involving multi-speaker speech and audio. Demos are available at the project page.