Streaming omni-modal models must decide what to answer, and when, based only on the video chunks and synchronized audio they have observed so far. According to a new paper, visual cues can often suggest an interpretation before an utterance or sound event has finished — and if that interpretation is stored as fact, later reasoning may keep repeating it even after the audio contradicts it.
The authors call this failure premature cross-modal commitment. Their proposed method, Omni-Streaming Thinking (OST), generates structured outputs containing evidence observed so far, forecasts of future evidence, and claims derived from that evidence. Each claim starts as pending and is tied to a future verification interval.
OST stores audio and visual evidence separately, then checks each claim against evidence from the specified modality at the end of its verification interval. When contradictory evidence appears, a refutation process reduces the influence of the claim and its dependent states, and guides a state update using the new evidence. An answer gate then decides whether answer-critical claims meet the conditions for responding.
Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST reportedly outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average.
The paper also introduces OST-DiagBench, which holds video fixed while editing audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d-prime = 2.95, versus at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.