Researchers have introduced OSWorld-Pro, a new evaluation framework designed to assess computer-use agents (CUAs) based on their procedural progress rather than solely their final deliverables. The work addresses a key limitation in existing benchmarks like OSWorld, where agents are judged only on end-state performance after hundreds of steps, offering little insight into how or why they fail.
The benchmark comprises over 300 tasks containing more than 2,800 subgoals, grounded in over 67,000 human annotations. To evaluate subgoal fulfillment, the researchers employ robust, human-aligned LLM-Judges that track progress through sequentially dependent subgoals.
Findings show that OSWorld-Pro is challenging even for state-of-the-art models. Top performers such as Claude Opus 5 achieved only 75.7% on OSWorld-Pro, compared to 83.4% on OSWorld. The study also identifies process-focused failure modes, including subgoal-irrelevant actions and click-based mistakes, which can inform targeted improvements in agent performance and efficiency.