Researchers at Hugging Face have unveiled OmegaUse-OfficeVal, a new benchmark designed to evaluate large language model (LLM) agents on long-horizon office-suite tasks with a focus on economic efficiency. The benchmark addresses a gap in existing evaluations, which often overlook the cost-effectiveness of AI agents in real-world office workflows.
The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners, adapted through a privacy-preserving process. On average, each task requires 2.32 hours of human labor to complete, providing a realistic baseline for comparison.
A key feature of OmegaUse-OfficeVal is its economic grounding: each task is paired with two economic signals—human labor time and a task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation, allowing for a nuanced assessment of agent performance.
To ensure stable evaluation, the researchers developed code-based verifiers from fine-grained rubrics. They evaluated several frontier LLMs alongside a human baseline. The results indicate that while all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality.
The code and dataset are fully open-sourced, with more information available on the project website: https://omegause-officeval.github.io.