In a new research paper, Hugging Face introduces SoftVTBench, a visuo-tactile dataset and benchmark designed to evaluate the physical interaction quality of deformable-object manipulation. Traditional benchmarks often measure only task success, ignoring issues like slippage or excessive compression that can occur even when a task is completed.
The dataset includes 4,000 expert demonstrations and more than 50 assets, including volumetric deformable objects and visually matched rigid twins. At 20 Hz, each episode synchronizes multi-view RGB, dual-finger tactile RGB and marker motion, proprioception, language, and binary and continuous gripper actions, alongside evaluator-only finite-element (FEM) states.
Building on this, the authors establish a closed-loop benchmark using fixed object-specific calibration to define the Deformation-aware Success Rate (DSR). A rollout is successful only if it completes the task and keeps peak normalized deformation within tolerance.
Across policies like Diffusion Policy, π0.5, and FastWAM, all 12 in-distribution configurations contained successful rollouts that violated deformation tolerance, accounting for 0.7–24% of each configuration's successes. Under distribution shift, visuo-tactile variants achieved higher task success in all six policy–suite comparisons and higher DSR in five, though in-distribution benefits were mixed.
The results indicate that providing tactile data alone does not guarantee effective multimodal fusion. SoftVTBench offers a common resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.