Researchers at Hugging Face have unveiled GE-Act 2.0, a novel world-action model designed for robotic manipulation. Unlike prior approaches that rely on pretrained video generators, GE-Act 2.0 initializes all its generative and action components from scratch, specifically on manipulation data. This allows the model to learn representations tailored to the control task, rather than inheriting generic visual features.
The architecture comprises three key components: a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). The CoAE retains action- and instruction-relevant information even under aggressive compression, while the SVP generates a complete future state in a single differentiable pass. This design enables separate pretraining of visual planning and inverse dynamics on complementary datasets, followed by joint training using a novel knowledge-aligned selective optimization (KASO) technique. KASO filters out mismatched supervision by selecting only predicted future states that are behaviorally compatible with the recorded action.
The model was evaluated directly from pretrained checkpoints, without any per-task fine-tuning, across 100 tasks spanning 20 manipulation skill groups, with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours improved success rates from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D. Notably, G2-90D, despite comprising less than 2% of the co-training data, improved by 17.7 points, suggesting effective cross-embodiment transfer.
Gains were observed across 19 out of 20 and 18 out of 20 skill groups for the two benchmarks, respectively. The authors found a strong correlation between skill-specific coverage and zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Furthermore, the model grounded object, color, shape, and position references in at least 90% of trials, and followed explicit instructions even when they conflicted with an already-committed behavior or a conventional scene association.