Researchers have released a technical report describing AuK, an open-source foundational model designed to unify speech generation and editing behind a single interface built on natural-language instructions and audio context.
According to the report, the model was trained on approximately 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision spanning five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing.
Architecturally, AuK combines a multimodal large language model for semantic conditioning, a VAE trained jointly on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer. That Transformer uses dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation.
Training proceeds in stages: a generation-only warm-up, then joint generation-editing pre-training, followed by post-training that includes human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation.
To cut inference costs, the team distilled the model using consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs four-step inference without classifier-free guidance and achieves a 4.5x wall-clock speedup over the full model under matched conditions, the report states.
The authors report leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. Source code and model weights are being released to support reproducibility and further research.