Large language model (LLM) agents are increasingly deployed as autonomous tool users, but most benchmarks evaluate them on bounded tasks with immediate success criteria. Real-world applications, however, often demand long-term coherence—the ability to maintain purposeful behavior over extended periods while adapting to accumulated evidence. This requires persistent environments where actions constrain future choices, feedback arrives with variable delays, and incoherent behavior leads to measurable cumulative effects.
To address this gap, researchers from Hugging Face and Alibaba introduce MerchantBench, a 365-day, order-level simulation grounded in 98,843 real e-commerce product records. The environment provides agents with 26 tools for product sourcing, listing and pricing control, cash-flow management, and mixed-latency feedback adaptation. Agents must navigate promptly observable upstream supplier events alongside delayed downstream order outcomes, such as refunds, negative reviews, and penalties, forcing them to follow individual order lifecycles and revisit earlier decisions.
The study evaluated eight LLMs under two agent frameworks across 48 year-long runs. The results reveal a substantial gap: the best LLM configuration achieved only 27.3% of the mean final net assets attained by human participants. Analysis shows that long-running does not necessarily mean long-horizon—agents may gradually stop acting, narrow their control loops, fail to respond to delayed feedback, or reinforce incorrect assumptions through memory.
MerchantBench is available with open-source code and a project homepage, providing a valuable resource for future research on long-term agent coherence.