A new research paper from Hugging Face, titled Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text, tackles a fundamental mismatch in how spatial reasoning is evaluated. Current benchmarks typically require text or coordinate outputs, which disadvantages image-generation models that naturally express spatial concepts through visual marks.
The authors propose ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics. ProVisE includes an Agentic builder that constructs and validates task-specific protocols for new benchmarks.
To support this evaluation, the team introduces SpatialGen-Bench, a curated diagnostic benchmark of 470 samples across 14 spatial subtasks, four capability levels, and diverse answer forms. They evaluate 31 text-output VLMs and image-generation models in a unified setting and validate Agentic protocol construction on six external spatial benchmarks.
Key findings reveal complementary strengths: image-generation models are competitive when spatial answers can be externalized directly in pixel space, while text-output VLMs retain a clear advantage in compositional spatial reasoning. Notably, GPT Image 2 correctly solves 37% of spatial cases missed by GPT-5.4, highlighting the value of pixel-based expression.
The paper establishes a metric-compatible testbed for studying spatial cognition in image-generation models, bridging the gap between visual and textual reasoning interfaces.