StepAudio 3 Gen is a general-purpose audio generation model described in a newly posted technical report. According to the authors, it supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, "vibe speech," and mixtures of multiple audio types within a single unified framework.
Architecturally, the model departs from the diffusion Transformer-based continuous generation approach common in recent general audio systems. Instead, it is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16×2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer retains both kinds of information.
For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis.
The report highlights three design principles behind the system: interference-aware progressive pretraining, which aims to add audio capabilities while preserving the language model's textual abilities; an RVQ Adaptor for incorporating multi-codebook acoustic representations; and discrete autoregressive modeling over a shared representation across general audio domains.
Following progressive pretraining, multi-task instruction training and supervised fine-tuning, the authors report state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects and music. Audio samples are available at stepaudiollm.github.io/step-audio-3-gen.