Computer-using agents (CUAs) are rapidly advancing, but evaluating whether they successfully complete tasks remains a challenge. A CUA trajectory records the agent's actions, states, and reasoning, and verifying task fulfillment is crucial for evaluation, data curation, and reinforcement learning. Human verification is not scalable, so the field increasingly relies on vision-language models (VLMs) as judges. However, the reliability of these VLM judges has been largely unexamined.
To address this, researchers introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms (web, Windows, Ubuntu, and mobile), with ground-truth verdicts established through multi-stage human annotation. The benchmark includes OSReward-Hard for challenging cases and OSReward-Multi for fine-grained efficiency and alignment scoring.
The most comprehensive evaluation of VLM judges to date reveals that even state-of-the-art models fall short of an ideal judge, exhibiting a systematic leniency bias that mislabels failed runs as successes. The few reliable models are too expensive for large-scale use, while affordable open models lag significantly behind.
To close this gap, the team constructs and releases OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments. They train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost. The code, benchmark, dataset, and model checkpoints are publicly available.