Researchers have released AgentWorld, a new benchmark designed to evaluate long-horizon collaboration among LLM-based multi-agent systems. According to the paper, existing multi-agent benchmarks tend to focus on competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, which the authors argue fails to isolate genuine collaboration capabilities.
AgentWorld comprises 100 human-annotated tasks, along with 100 augmented variants, set within a rich MMORPG sandbox. Tasks span more than 50 interaction rounds and require between 3 and 20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing. The benchmark operates under a blackbox setting in which each agent acts independently without access to others' internal states.
Beyond conventional binary task success, the authors propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome.
Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success. The paper reports systematic failure modes including communication breakdowns, role confusion, and an inability to maintain shared plans across rounds.
AgentWorld is described as fully open-source.