Hugging Face researchers have released CLBench-V, a comprehensive benchmark designed to evaluate how well multimodal AI models can learn from context—a capability that goes beyond relying solely on pre-trained knowledge. The benchmark addresses a critical gap: existing evaluations focus primarily on textual contexts, whereas real-world tasks often involve multimodal inputs such as figures, tables, maps, and web pages.
CLBench-V organizes tasks along three dimensions: context grounding (linking information to specific parts of the input), new information application (using novel data to solve problems), and new knowledge learning (acquiring entirely new concepts from context). The benchmark combines converted public datasets with newly constructed ones spanning science, finance, long-document understanding, spatial reasoning, and web-based visual question answering.
To reduce construction costs, the team developed automated procedures for generating and filtering domain-specific context-learning tasks. The final benchmark comprises 3,443 instances tested across six recent multimodal models. Results show that the best overall score is only 0.2847, indicating that multimodal context learning is far from saturated.
Among the models evaluated, InternVL3.5-30B-A3B performed best on context grounding and new knowledge learning, while Qwen3.5-Plus excelled at new information application. The paper also analyzes judge reliability, context length, image count, and representative failure cases. Code and datasets are available on GitHub.