In the realm of embodied AI, a conversational avatar must decide both what to say and how to move while speaking. Traditionally, these capabilities have lived in separate model families: spoken dialogue models generate speech without motion, while co-speech motion models generate motion only from pre-existing audio. The common solution—a cascade that first generates speech and then runs a motion model over the finished audio—requires a second full inference pass and prevents joint optimization between the two processes.
To address this, researchers at Hugging Face present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expressions along with hand, upper-body, and lower-body motion, all generated directly from the hidden states that produce the speech. This joint training is critical: with the speech pathway frozen, motion remains misaligned with the audio, but co-adapting the LLM, Speech Generator, and Motion Generator under both objectives restores alignment while preserving spoken-dialogue ability.
The training pipeline leverages a scalable, model-agnostic pseudo-labeling approach that pairs consistent-voice speech responses with a replaceable motion teacher, yielding a dataset of 422,856 quality-ranked pairs (1,402 hours). The team also releases SwDA-500 and, to their knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue. This protocol matches audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement.
Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4× faster (RTF=0.78, faster than real time). It surpasses all non-teacher cascades on beat correlation and diversity, and achieves a 2.62% word error rate—the lowest among the omni-modal systems compared.