Hugging Face has released Open-AoE, an open, community-oriented egocentric manipulation dataset and toolchain designed to bridge the gap between human video capture and robot learning. The project aims to provide scalable supervision for embodied intelligence by combining low-cost continuous capture with structured annotations and reusable tools.
The first release includes approximately 2,000 hours of manipulation video collected in natural environments by over 500 contributors using more than 400 smartphones. The dataset offers text annotations, MANO-based hand poses, camera trajectories, and temporally localized atomic actions, enabling detailed analysis of human manipulation.
Open-AoE features a comprehensive data processing pipeline that transforms raw recordings into structured samples through temporal action segmentation, semantic annotation, hand reconstruction, and camera trajectory reconstruction. Additionally, a separate downstream toolchain supports visualization, cross-embodiment retargeting, model-specific data conversion, and training recipes for VLA policies, WAMs, and World Models.
By integrating scalable capture, structured processing, and downstream adaptation, Open-AoE reduces barriers to data contribution and reuse, providing practical open infrastructure for embodied model training, human-to-robot transfer, and world modeling. The project aims to accelerate exploration in embodied AI by openly releasing datasets, toolchains, and continuously updated training methods.