Researchers have introduced Tri-PvP, a new benchmark designed to expose modality bias in omni-modal large language models (OLLMs) — systems that jointly process vision, audio, and text. The work addresses a gap in existing evaluations, which conflate two distinct forms of evidence within a single modality: perceptual signals (such as a photograph or recording of a dog) and propositional signals (such as the declarative claim "this is a dog").
According to the paper, this conflation means that measured modality bias is inherently confounded with evidence-form bias, making it impossible to attribute behavior cleanly to either source. Tri-PvP tackles this with an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, where vision and audio each take either perceptual or propositional form.
Evaluating five OLLMs, the researchers found robust visual bias across most models and evidence-type conditions. More notably, they uncovered a systematic asymmetry: models exhibit a stronger bias toward perceptual signals in vision but toward propositional signals in audio.
Further analyses using layer-wise linear probing and contrastive decoding showed that modality bias is already linearly decodable from early representation layers and can only be partially mitigated. The authors argue this calls for mitigation strategies that go beyond surface-level interventions.
The benchmark and code are publicly available, with the dataset hosted on Hugging Face and the code on GitHub.