Hugging Face researchers have introduced Ego2Robot, a scalable pipeline that transforms egocentric human manipulation videos into robot training data. The approach addresses the need for large-scale, diverse demonstration data to train generalizable robot manipulation policies.
While previous work showed that retargeting and rendering such videos into robot-format data works for per-task policies at small scale, Ego2Robot is the first to explore its pretraining benefits for vision-language-action (VLA) models at scale. The pipeline includes action retargeting, robot-arm visual synthesis, and multi-level quality curation, supporting both curated datasets and in-the-wild videos.
The result is the largest ego-to-robot dataset to date: 18,561 hours of robot training data spanning 15 robot morphologies. To evaluate generalization, the team extended RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics.
Experiments show that joint pretraining on Ego2Robot-synthesized and real robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits confirmed in real-robot deployment. The project page is available at https://www-ye.github.io/ego2robot_blog/.