The Human Engineering Research Laboratories (HERL) at the University of Pittsburgh, in collaboration with ATDev, is developing the Robotic Assistive Mobility and Manipulation Platform (RAMMP) with up to $41.5 million in funding from the Advanced Research Projects Agency for Health (ARPA-H). The project aims to address the needs of an estimated 5.5 million wheelchair users in the United States, where over 100,000 wheelchair-related injuries occur annually.
RAMMP integrates advanced robotics, novel operating systems, digital twin technology, and artificial intelligence, including Meta's open-source AI vision models DINO and Segment Anything Model (SAM). DINO, a self-supervised vision transformer, excels at learning visual representations from unlabeled data, while SAM can identify and outline any object in an image or video with minimal prompting.
Deploying these models on edge devices—processing data directly on the robot rather than in the cloud—enables real-time perception and response to dynamic environments. The team optimizes DINOv3 and SAM for battery-powered hardware, balancing precision with speed and reliability. According to Sivashankar Sivakanthan, Chief of Staff for RAMMP, performance is measured by reliable operation in everyday life, not just benchmark accuracy.
The perception system uses RF-DETR, a lightweight detection model fine-tuned with DINOv2 embeddings, with training data auto-labeled using SAM. This allows rapid generation of high-quality annotations across various angles, heights, and lighting conditions, achieving real-time 360-degree environmental awareness.
The RAMMP consortium includes partners like Kinova Robotics, LUCI Mobility, ATDev, and academic institutions such as Carnegie Mellon, Cornell, Northeastern, and Purdue. The project aims to create a future where individuals with limited mobility can live more independently, with Meta's vision models playing a critical role in helping the robot understand scenes and navigate the world.