Zero-shot cross-task generalization—where a robot must execute manipulation tasks never seen during training—remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This in-context learning (ICL) paradigm turns generalization into a problem of task specification.
To bring this paradigm to robotic manipulation, researchers from Hugging Face and collaborators present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. The key insight is that human videos provide richer visual cues about the intended task evolution than language alone.
To address the scarcity of task-rich paired human-robot data, the team developed an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos. This yielded HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks.
For training, they introduced an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieved a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline.
In real-world evaluations, Zero-WAM followed human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion. The project website, code, and paper are publicly available.