Recent advances in omni-modal large language models (Omni-LLMs) have opened the door to real-time video assistants that can perceive environments and guide users through multi-turn conversations. However, evaluating these assistants is challenging because their unpredictable responses dynamically alter user actions, making static datasets inadequate.
To address this, researchers at Hugging Face introduce OmniAssistBench, a benchmark that reverse-engineers existing Internet videos to simulate continuous interactions. The pipeline deduces logical user goals and segments videos into multi-turn clips, requiring over 1,000 expert person-hours to build.
In tests, proprietary Gemini-3-Pro scored 66.4 out of 100, while open-source Qwen3-Omni-Instruct achieved 51.2. Although models generally understand user inputs, they often provide incorrect or incomplete answers, struggling with visual prompts like hand gestures, failing to maintain historical context, and responding prematurely before target events.
The findings indicate substantial room for improvement before Omni-LLMs can become reliable real-time assistants.