Agent benchmarks have traditionally focused on bounded tasks with clear stopping points, but real-world operations like running a business are open-ended. To address this, the Qwen team at Alibaba, in collaboration with Taobao & Tmall Group, has released E-Commerce Bench, a new benchmark designed to measure long-horizon operational ability in a realistic e-commerce environment.
The benchmark places an AI agent in the role of an online merchant, starting with ¥100,000 in capital and the ability to run multiple stores. Over a simulated 365-day period, the agent must make daily decisions: researching categories, negotiating with suppliers, pricing and listing products, managing promotions, and handling inventory and cash flow. The environment is built on real, desensitized Taobao & Tmall data, featuring 6,886 products across 60 categories, 576 suppliers, and 12 store types, with 10 market events and 8 promotions on the calendar.
To ensure reproducibility, the benchmark uses deterministic models for customer demand and supplier behavior. Customer demand is computed from named factors without sampling, while supplier negotiations are driven by a deterministic kernel that generates quotes and concessions, with an LLM only rendering the dialogue. This prevents inconsistent outcomes and avoids the risk of jailbreaking the supplier to get unfair prices.
Results from testing 18 models over five episodes each show dramatic differences. GPT-5.6 Sol finished with ¥1.43M (14.31x its stake), while Qwen3.5-Plus averaged just ¥1,100 left. The strongest open-weight model, Qwen3.8-Max-Preview, achieved 4.16x, but the top four overall were closed-source. Ten of 90 episodes ended in bankruptcy, often due to overstocking early in the year.
Beyond year-end assets, the benchmark scores six additional dimensions: negotiation quality, fraud avoidance, cash flow and solvency, operational efficiency, operations execution, and learning over the horizon. Notably, fraud avoidance showed the widest divergence, with fraudulent procurement spend ranging from 0.12% for Claude Opus 4.7 to 20.11% for Qwen3.5-Plus. The benchmark aims to provide a more holistic evaluation of an agent's ability to operate in dynamic, long-horizon settings.