Researchers have introduced BeyondSCe, a zero-shot robotic grasping system designed for what the authors call the event-referential setting. In this scenario, a robot observes people interacting with objects and is later asked to act on a target specified by the role it played in a past event, rather than by its name or appearance.
A key challenge is that the requested object may no longer be visible when the robot is asked to grasp it. To address this, BeyondSCe takes the event history and the current scene as input, identifies the requested object or part, and localizes it for grasping.
When the target is occluded, the system combines an event prior recovered from the history with current scene geometry to select camera viewpoints likely to reveal the target. According to the paper, the system relies on pretrained models without any additional task-specific training.
In real-robot experiments using a single wrist-mounted RGB-D camera, BeyondSCe reported grasp success rates of 76% for initially visible targets and 77% for occluded targets, compared with 40% and 55% for the strongest baseline in each condition.
On four additional scenes with heavy occlusion, the system increased grasp success rates from 75% to 95% while reducing the mean number of views from 3.35 to 2.20, compared with an active-perception baseline that was given the target's ground-truth 3D bounding box.