A newly posted paper on Hugging Face defines OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In this setup, the model receives audio and video from the user simultaneously and returns text, with the user's query embedded in the audio and video rather than supplied as a separate text question, external captioning, or speech recognition.
The authors argue that direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, they note that research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce, and a good reply often depends on the user's surroundings, facial expressions, and nearby objects, which can be expressed in many ways, making keyword matching unreliable for judging reply quality.
To address this, the paper presents OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. The synthesized dialogues are used to build OmniVChat-Bench, an evaluation benchmark covering five ability categories of basic dialogue ability.
The work also introduces OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style. According to the paper, training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human, validating the reward design and showing transfer to real-world dialogues in training and evaluation.
The paper is accompanied by a Hugging Face dataset repository and a GitHub repository.