Robot learning increasingly depends on broad and diverse demonstrations, yet collecting robot data remains expensive and poorly suited to covering the long tail of real-world tasks. To address this bottleneck, researchers introduce RoboTok, an internet-scale data engine that, given a query human manipulation video, retrieves manipulation-relevant human demonstrations from web videos for training dexterous robot policies.
The core innovation is a latent motion space learned from 3D hand trajectories expressed in estimated actor-centered reference frames. This representation enables manipulation behaviors to be compared across variations in camera viewpoint, scene appearance, and actor occlusions, while remaining compact enough for efficient search and continual indexing over internet-scale video collections.
Evaluations against existing robot-data retrieval approaches show that RoboTok retrieves more relevant manipulation demonstrations and improves downstream task success. This establishes hand-pose trajectory-aware retrieval as a scalable and continuously growing source of supervision for robot learning.
The project is open source, with code, data, and models publicly available. For more details, visit the project site, paper, code, and data/models.