Hugging Face researchers have introduced CORE, a novel method that enhances compositional reasoning in multimodal large language model (MLLM) embedding models. The approach distills fine-grained ranking judgments from a cross-attentive reranker into an embedding model, addressing a key limitation where embeddings struggle to distinguish scenes with identical concepts but different attribute-object bindings.
The method synthesizes candidate lists spanning five compositional matching levels and employs a Rank-KL objective to train the embedding model to replicate the reranker's ranking. The researchers also developed a graded evaluation protocol to compare contrastive learning, pairwise CoSENT, and listwise Rank-KL under identical data and tuning budgets.
Results show that both CoSENT and Rank-KL leverage multi-level supervision more effectively than contrastive learning, with Rank-KL achieving the strongest overall performance. Across three compositional reasoning benchmarks—COLA, SUGARCREPE++, and NEGBENCH—CORE-RERANKER-8B achieves an 82.7% total average, outperforming Jina-Reranker by 10.7 points. Meanwhile, CORE-EMBED-8B achieves the best total average (0.666) among all evaluated embedding models.
The improvements also transfer to the MCMR benchmark without sacrificing retrieval performance on COCO and Flickr30K. The team has released new embedding models that excel in fine-grained multimodal retrieval and compositional reasoning tasks.