Dense retrievers have advanced considerably on text and image corpora, but a new paper posted to Hugging Face asks whether those gains hold when a corpus mixes text, images, and fused text-image documents. The authors report that retriever performance is highly sensitive to modality composition.
As image documents are progressively replaced with semantically corresponding text representations, retrieval quality follows a pronounced V-shaped curve: it stays strong on single-modality corpora but degrades substantially once modalities coexist. Notably, irrelevant text causes more severe degradation than an equal number of irrelevant images — a phenomenon the authors term Chaos in the Text.
Further analysis points to a modality preference: text representations receive systematically higher similarity scores, which allows irrelevant text to outrank relevant images.
To mitigate this bias, the authors introduce Trident, which builds text, image, and fused text-image views of each document as co-equal positives and jointly optimizes relevance discrimination and positive-view balance through Multi-Positive View InfoNCE.
Experiments across visual document and natural image benchmarks show Trident improves mixed-modality retrieval on both CLIP-based and VLM-based architectures, reduces sensitivity to modality composition and text distractors, and increases average single-modality retrieval performance.