Researchers have introduced RoboDawn, a human-intuitive interface that exposes robotic control to an agentic vision-language model (VLM) through a compact set of discrete commands for translation, rotation, and gripper operation. The work investigates whether the intelligence of VLMs can transfer from the digital world to the physical world for robotic control.
Using this interface, the VLM operates in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions based on the resulting state. The authors also propose an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies.
Experiments on RoboTwin 2.0 C2R and RoboDojo show that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, it outperforms several strong policies trained on benchmark-specific robot data. A single in-context demonstration further boosts performance to state-of-the-art levels.
On RoboTwin 2.0 C2R, the success rate rises from 53.2% zero-shot to 73.6% one-shot, exceeding the baseline π0.5 (46.0%). On RoboDojo, success improves from 35.67% zero-shot to 47.17% one-shot. The framework also transfers to real-world robots, performing block-in-basket and block stacking tasks on a Franka arm.
The authors highlight three key findings: strong performance on the challenging RoboDojo benchmark, the effectiveness of in-context learning for embodied agents, and a natural test-time scaling effect where relaxing reasoning or interaction step limits continuously improves performance.