Researchers have released Argo-Bench, a new evaluation framework designed to test how well AI agents handle enterprise-scale data science and analytics workflows rather than simple query generation. The work, published as a preprint on Hugging Face, argues that existing text-to-SQL benchmarks are insufficient because they evaluate query writing alone and have been shown in audits to contain frequently incorrect answer keys.
According to the paper, real enterprise warehouses are too sensitive to release publicly, so prior benchmarks rely on public datasets where a single business event fits into one table. Argo-Bench instead simulates a food delivery platform in New York City at true scale, drawing on public data, peer-reviewed industry literature, and regulatory filings. The simulated world includes 81 million orders in 2024, grounded economics, fraud patterns, and marketplace incentives.
This world is exported to an ERP warehouse of 235 tables and 7.5 billion rows, modeled on the Oracle E-Business Suite schema. Crucially, the simulator's ground-truth state is withheld from the warehouse the agent sees, so tasks require reconstructing facts by navigating the warehouse before acting on them. The benchmark comprises 210 data science and analytics tasks, each with an executable reference solution demonstrating solvability using only the warehouse.
Unlike text-to-SQL benchmarks, Argo-Bench asks agents to take actions such as banning fraudulent accounts, allocating courier incentive budgets, or issuing back pay. A grader then scores each action by its consequences in the simulator. The first author noted that most established benchmarks do not yet cover the decisions companies care about most, and that while some micro-level benchmarks exist, none had tested day-to-day decision-making across areas like forecasting or fraud prevention at very large companies.
Results were sobering: among 14 frontier and open-weight models tested, the strongest scored 95 or higher on only 34.8% of tasks and averaged 59.5 points. The authors hope Argo-Bench will drive progress toward agents that can understand, navigate, and act within real data environments.