Estimating where a person is looking from images in real-world settings has long been a challenging task. Traditional methods rely on multi-stage pipelines that require explicit inputs such as head bounding boxes and human pose, making them susceptible to cascading errors. Moreover, they lack the flexibility to specify the subject of interest through natural language, a capability that has proven highly convenient and scalable in other image analysis domains.
To address these limitations, researchers from Hugging Face and collaborators introduce the Promptable Gaze Target Estimation (PGE) task. This new paradigm conditions gaze prediction on flexible user prompts—either textual descriptions like "the boy in the red shirt" or visual cues such as "person in point [0.52, 0.48]"—to identify the specific subject for gaze analysis. By integrating subject localization with gaze estimation, PGE eliminates the rigid dependency on intermediate analysis stages.
To support this new task, the team developed a scalable data engine that generated Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark comprising 120,000 high-quality, prompt-annotated image pairs. They also propose GazeAnywhere, the first model designed specifically for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders, simultaneously solving subject localization, in/out-of-frame presence, and gaze target heatmap estimation.
GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem, even on a challenging out-of-domain, real-world clinical dataset. The model is open-sourced and available at github.com/IrohXu/GazeAnywhere. The work has been accepted at CVPR 2026.