Hugging Face researchers have introduced VibeVoice-ASR-Streaming, a novel end-to-end model that brings speaker-attributed automatic speech recognition (ASR) to real-time applications. Unlike traditional systems that treat ASR and speaker diarization as separate tasks, this model unifies them in a single LLM-based architecture, enabling low-latency transcription of "who said what" as speech arrives.
The model interleaves fixed-size audio chunks with a small amount of lookahead audio and previous text, allowing it to generate speaker-attributed transcriptions without a separate diarization stage. This design addresses the limitations of existing unified models, which primarily support offline recognition and fail to meet the low-latency demands of real-time voice assistants and agents.
In evaluations, the 7B parameter version of VibeVoice-ASR-Streaming achieved the lowest average word error rate (WER) and character error rate (CER) across five evaluation sets. For speaker attribution, it achieved the best or tied-best performance on 12 of 13 evaluation settings. The researchers have released the 1.5B and 7B model weights along with inference code.
This work represents a significant step forward in streaming speech processing, potentially enabling more natural and efficient interactions with voice-based AI systems.