Vision-language models (VLMs) are increasingly being used as agents that act on 3D scenes, but existing benchmarks often limit evaluation to textual responses or single-object operations. To address this gap, researchers from Hugging Face have released SceneActBench, a new benchmark designed to assess how well VLM agents perform visually conditioned actions in complete multi-object 3D environments.
The benchmark covers five distinct 3D tasks, built from 210 source instances that yield 520 task cases, including paired input conditions. Each task runs through a fixed agent-environment loop to ensure fair comparison across models. Agents receive PNG images or sampled video frames (and, where applicable, 3D assets) and must act on the 3D scene. Their outputs are evaluated against hidden ground truth using task-specific geometric metrics.
In tests across eleven proprietary VLM configurations, overall scores ranged from 38.6 to 50.2. Notably, no single model performed consistently well across all tasks, highlighting the challenge of general-purpose 3D scene understanding and action. The paper also includes an analysis of where and how failures occur, providing insights for future improvements.
SceneActBench is available on Hugging Face Papers, offering a new tool for the community to benchmark and advance VLM agents in 3D environments.