Embodied agents may look far more capable of following instructions than they actually are, according to a new research paper introducing RoboFollow, a diagnostic benchmark for instruction-conditioned manipulation. The authors attribute this gap to a structural property they call low scene entropy: when a visual scene permits only one valid task, language becomes redundant, and a policy can post high success rates while barely relying on the instruction at all.
To counter this, RoboFollow is built on three principles. First, high scene entropy: each training scene supports multiple kinematically distinct task branches, so vision alone is insufficient and the policy must use language. Second, a hierarchical diagnostic protocol spanning four levels (L0–L3) that progressively perturbs visual layout and semantics, testing whether equivalent instructions produce consistent behavior and distinct ones produce discriminable behavior across spatial relations, attributes, trajectory constraints, and logic. Third, confound-controlled diagnosis: interaction objects are simplified, actions are restricted to the trained repertoire, and stage-wise Intent and Execution scores are reported to separate comprehension from motor execution.
The benchmark is a simulation environment built on RoboTwin, providing four scene families, expert demonstration collection, the L0–L3 evaluation protocol, and the stage-wise Intent and Execution scores. Multiple tasks share the same scene configuration, forcing policies to rely on language to select the right object, spatial relation, motion constraint, or logical branch.
Evaluating nine VLA and WAM policies, the authors report that strong L0 performance, where achieved, does not reliably transfer to the L1–L3 levels under their fine-tuning setup. Representative mitigations — stronger VLM backbones, QA co-training, LangForce, and Classifier-Free Guidance — all failed to close the gap.
The paper concludes that genuine instruction following remains a critical and overlooked bottleneck for embodied agents. Code and dataset are released publicly, with the paper, project page, GitHub repository, and Hugging Face dataset linked by the authors.