In a new research paper, Hugging Face introduces EASEL, a benchmark designed to evaluate a critical yet under-explored capability in multimodal AI agents: dexterous visual tool use. This refers to fine-grained, closed-loop actions where models must infer tool parameters from visual evidence, and those parameters directly determine the final output.
The benchmark uses reference-guided visual reconstruction as its primary task: an agent incrementally paints a canvas to match a reference image. EASEL also includes semantic tasks such as region annotation, handwriting, and path planning. To support training, the authors provide EASEL-Data, a 440k-sample curriculum dataset, and EASEL-9B, a model trained on this data.
Evaluating 25 models, the study finds that current multimodal agents systematically struggle on EASEL. Reconstruction similarity scores remain low, between 0.40 and 0.54, and trajectory diagnostics reveal severe closed-loop instability, with models often saturating early or degrading after reaching a peak. Semantic tasks highlight sharp capability boundaries in precision annotation and path planning.
EASEL-9B, trained on EASEL-Data, surpasses its base model by a relative 6.3%, ranking third among all evaluated models. The results underscore the need for better supervision and evaluation methods for agentic visual tasks.