A recent research paper from Hugging Face, titled "The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images," challenges the effectiveness of visual tool-use in multimodal large language models (LLMs). The study argues that although these models often show improved aggregate accuracy when using visual operations like crop-and-zoom, the returned observations frequently fail to causally influence the final answer, or are used incoherently.
The researchers formulate visual tool-use as a causal graph, separating observation-mediated paths from action-induced shortcuts. They audit this framework through three levels of intervention: policy-level (comparing tool-use with direct inference), trajectory-level (corrupting all observations during rollout), and step-level (counterfactually replacing individual observations). A key contribution is the step-level estimand called Visual Evidence Gain, which isolates the causal contribution of each returned observation.
Across six representative models and five fine-grained perception benchmarks, the audit uncovers two primary failure modes: Calling Without Looking, where returned observations have no causal effect on the answer, and Looking Without Planning, where observations are informative but the call schedule is incoherent. The trajectory-level diagnostic shows that the policy-level accuracy gain is concentrated in a small, 'Calibrated' minority of rollouts.
The authors term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code for the audit is available on GitHub at OpenCausaLab/CauAudit.