Multimodal large language models (MLLMs) increasingly use sketches, annotations, and intermediate images during reasoning, but whether they genuinely depend on these visual states has remained unclear. Existing benchmarks often suffer from narrow task coverage, partially text-solvable samples, and evaluations that focus only on final answers without diagnosing the generation, rendering, and use of intermediate visual states.
To address this gap, researchers at Hugging Face introduce See2Think, a unified evaluation framework comprising two components: See2ThinkBench and Visual Action-of-Thought (VAoT). See2ThinkBench includes 1,200 open-ended, visually dependent problems across 12 task categories spanning 2D structured, 3D scene, and real-world reasoning. VAoT records textual thoughts, visual actions, rendered states, and subsequent reasoning under four controlled inference settings.
Evaluating representative proprietary and open-source multimodal models, the study finds that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks. Process analysis reveals that models usually select relevant visual operations, but faithful rendering remains the clearest bottleneck, and high feedback uptake does not necessarily translate into accuracy gains.
Under task-relevant corrupted feedback, models exhibit behavioral dependence on visual states, with accuracy dropping by over 10 percentage points in controlled interventions. The authors hope See2Think provides a new perspective for evaluating and improving visual reasoning capabilities in multimodal AI systems.