Researchers have proposed RLHND, a hand tracking model built on a video foundation model that jointly estimates hand pose and realistic tactile information from monocular egocentric videos. The work addresses a common limitation in robot learning from human video: most existing hand trackers regress pose from cropped frames with weak priors on hand motion and hand-object interaction, producing estimates that are inaccurate and physically inconsistent.
The method converts the pre-trained Cosmos 3 video diffusion backbone into a deterministic clip-level feature extractor through clean-latent conditioning, carrying its learned priors on hand motion and hand-object interaction into the tracking task. For pose estimation, RLHND predicts hand poses with anatomically plausible joint angles and supports optional conditioning on the shape parameter, which helps maintain a consistent hand shape within a video and even across videos recorded by the same actor.
For tactile estimation, a separate tactile expert stream is trained while the pose stream is frozen, predicting dense contact and force over the hand surface. The authors also adopt LBS-based feature spreading to enable vertex-wise feature extraction without costly per-vertex attention.
According to the paper, RLHND achieves state-of-the-art performance across various benchmark datasets for pose estimation, and also reaches state-of-the-art performance in contact and force estimation. The authors further demonstrate the utility of RLHND for robot learning through retargeting results and real-world robot experiments.
The code is slated to be publicly available at the project page linked in the paper.