Multimodal entity linking connects mentions in text and images to entries in a knowledge base. These systems often struggle with rare entities, but previous work has measured rarity mainly through popularity metrics such as pageviews. A new paper broadens this view by using knowledge-graph structural metrics that capture how well an entity is documented and connected, revealing many rare entities that popularity metrics miss.
Across the resulting rare-entity slices, state-of-the-art accuracy drops by 15.4–39.9%, showing that different definitions of rarity expose different failure modes. To address these failures, the authors introduce a simple, training-free framework in which a reasoning-capable vision-language model iteratively searches and reasons over Wikipedia, gathering evidence dynamically.
Controlled experiments show that reasoning and retrieval are complementary. Reasoning alone does not significantly improve accuracy on rare entities. Retrieval without reasoning improves rare-entity accuracy but can hurt overall accuracy. Their combination performs best.
On MERLIN, a multilingual multimodal entity linking benchmark covering five languages (Hindi, Indonesian, Japanese, Tamil, Vietnamese), the best system improves over the state of the art by 6.9% overall and by up to 23.3% on rare-entity slices. The authors release MERLIN-Rare, rare-entity test slices for targeted evaluation, along with their framework.
The release includes 1,105 mentions across five languages, augmented with 15 rarity metrics, complete model predictions, and reasoning and search traces.