Researchers have introduced OmniSeek, an agentic framework that converts an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use. The work reframes audio-visual understanding as an evidence-acquisition problem rather than a single forward pass over an entire sequence.
Under the OmniSeek protocol, the model dynamically decides whether to look or listen, and over which temporal window, in order to retrieve sparse but critical evidence across modalities within long contexts. Retrieved raw audio or visual segments are appended back into the context through an iterative multi-turn loop to support subsequent reasoning.
To cold-start this capability, the team built a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with interleaved audio and visual evidence. The model is first supervised on these trajectories to instill multi-turn tool-use behavior, then optimized with a two-stage reinforcement learning procedure using verifiable rewards.
The authors also propose an Audio-Visual Necessity objective that explicitly rewards successful trajectories whose reasoning depends on both modalities, discouraging single-modality shortcuts. According to the paper, extensive experiments across a wide range of benchmarks show that OmniSeek learns adaptive cross-modal evidence seeking and consistently improves audio-visual reasoning performance.
The paper notes that code, weights and data are under Adobe's internal review.