Knowledge-based visual question answering (KB-VQA) often requires retrieving external information to answer questions about rare or long-tail entities. However, traditional retrieval systems typically rely on CLIP-style dual encoders that match images based on surface-level visual similarity, which can fail when identical concepts look very different or when distinct entities appear visually alike.
To address this, researchers from Hugging Face and collaborators propose KBMR, described as the first multimodal language model (MLLM)-based embedding retriever tailored for KB-VQA. By leveraging the autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity, moving beyond simple visual appearance.
A key innovation is an MLLM-based semantic discriminator that generates continuous entity-consistency weights, guiding a novel continuous semantic distillation objective. This enables effective hard negative sampling and soft supervision, replacing rigid binary labels that often introduce noise in Wikipedia-scale retrieval.
Experiments on E-VQA, InfoSeek, and OK-VQA show that KBMR significantly outperforms CLIP baselines, with up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. The code is publicly available on GitHub.