Researchers have published a new paper on Hugging Face describing Agent as Policy (AGP), a framework that lets a general-purpose agent directly drive a physical robot throughout task execution. According to the paper, the approach requires no task-specific or environment-specific training.
Under AGP, task planning and execution are placed under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. The model weights remain fixed while the agent's reasoning and programming capabilities interact continuously with the physical world.
The authors evaluated AGP across multiple real-world manipulation tasks, including assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. Reported results include success rates of 100%, 100%, and 80% on three block construction configurations, and at least 8 successful trials out of 10 for each evaluated configuration in assembly, block construction, and dice flipping.
The paper also notes that reusing saved procedures and programs reduces execution time across repeated trials. Agent trajectories are available on Hugging Face, alongside a short video showing AGP on real robot tasks and how the agent controls the robot through tool calls and visual feedback.
The authors frame these findings as a path for general-purpose agents to act as robotic policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.