Researchers have released EgoTools, described as the first comprehensive suite for understanding tool use from an egocentric, or first-person, perspective. The work targets embodied tasks in which agents must act under physical constraints while tracking changing object and task states, a setting where tool use is central and requires reasoning about affordances, hand-tool-object geometry, procedural progress, and causal effects on target objects.
According to the paper, current multimodal video models perform well on perception-oriented tasks such as captioning and general video QA, but remain limited in tool-centric embodied reasoning. Progress has been constrained by a lack of real-world egocentric data and diagnostic benchmarks.
EgoTools addresses this with two components. EgoTools-Data is a large-scale corpus of 100 hours of tool-centric egocentric recordings with synchronized audio, dense captions, reasoning-heavy narrations, and supplementary 3D information. EgoTools-Bench is a diagnostic benchmark of 1,000 QA pairs across four tracks, spanning tool-use understanding from perception and geometry to procedure and causal reasoning.
Experimental results indicate that models still struggle to ground tool use in visual evidence. Gemini-3.1-Pro reaches 66.9% overall accuracy but only 51.7% on the Perception & Grounding track. The authors also validate EgoTools-Data as a training resource: on the full 1,000-question benchmark, full supervised fine-tuning improves Qwen3-VL-8B-Instruct from 50.0% to 60.9%, under strict source-video separation.
The researchers position EgoTools as a unified resource for both training and diagnostic evaluation of real-world egocentric tool-use understanding. Project page, code, and dataset links are provided alongside the paper.