Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. To address this, Hugging Face researchers have introduced Ex-Omni-2D, a framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video.
Given a multimodal query, reference image, and reference audio, the model predicts a structured Visual Thought Plan (VTP) describing scene, emotion, and motion. This plan guides the generation of response text and native multi-codebook speech units, which form a shared acoustic-temporal interface. These units are decoded into speech and aligned online with video frames, enabling the response and avatar pathways to be learned from heterogeneous data sources without requiring large-scale query-text-speech-video supervision.
The framework employs a full-sequence Video Generator as the primary teacher, which is distilled into a few-step block-causal Streaming Student. The student's Prefix Streaming mechanism carries a clean latent across consecutive chunks, reducing cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end real-time factor (RTF) of 1.293 at 400×720/720×400 resolution, offering a practical quality-efficiency operating point.
The researchers invite feedback on visual presence, streaming generation, and the future of embodied dialogue systems, highlighting the potential of giving AI models a visible presence in conversations.