Researchers publishing on Hugging Face have introduced ModAR, described as the first world-action model (WAM) that autoregressively denoises multiple future modalities before predicting actions. The work, titled Modality-Autoregressive World-Action Models, addresses an open question in the field: how best to combine visual modalities such as depth, pretrained visual features, and point tracks within WAMs, which typically predict the future as RGB images.
ModAR's sequential generation approach allows each prediction to condition on previously generated modalities. The authors trained the model from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance.
According to the paper, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. The authors also report that ModAR's sequential generation outperforms existing WAM formulations, achieving the highest average success rate at all evaluated data scales.
In a comparison with Flex-π, a video-model-initialized WAM fine-tuned on the same data, ModAR achieved a slightly higher observed average success rate of 75% versus 72% while using approximately 20 times fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR reportedly outperforms baselines and improves with human videos.