Researchers have proposed World Action Agent (WAA), a multi-agent harness that lets general-purpose vision-language models (VLMs) pilot robots using basic tools, with every decision made inside a visual action workspace. The work, posted on Hugging Face, argues that existing systems either use VLMs indirectly — to predict constraints or write programs — or merely give them a view of the scene rather than a world in which to act.
The workspace rests on three properties. Contact views are selected automatically from scene geometry to present the area around the current interaction. Action rehearsal turns each action into an editable proposal that the agent previews and revises against planning feedback before execution, either alone or through an Imagination Agent. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them.
WAA also acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, while its interaction traces train smaller VLMs to pilot the same harness.
On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone. The same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.