AI agents increasingly operate in persistent environments where early actions can have cascading effects on future decisions. Unlike standard language-model interactions, agent behavior is mediated through a shared state that is repeatedly modified across long-horizon workflows. Current safety benchmarks, however, often focus on short, static tasks and fail to capture these cumulative risks.
To address this gap, researchers from Hugging Face introduce OpenART, an open-ended arena for scalable agent red teaming through environment evolution. OpenART provides over 10,000 validated stateful scenarios across 50 domains, drawing from a pool of more than 500,000 tools and skills. These tasks require a median of 97 tool calls and enable unified evaluation across 75 different agent-model configurations.
The team also proposes the Evolutionary Markov Hypergraph Attack (EMHA), a black-box policy that performs feedback-driven environment evolution by coordinating authorized state transitions without requiring parameter updates. Throughout the evaluation, task objectives remain fixed while only the environment state changes.
Across all configurations, EMHA achieves a pooled Attack Success Rate (ASR) of 85.0%. Its advantage over instruction-only evolution increases from approximately 2% on simple environments to over 17% on the most complex ones, demonstrating that environment evolution increasingly exposes safety failures as task complexity grows. Furthermore, the analysis shows that the specific runtime implementation of an agent explains a significant portion of safety variation beyond the underlying model's capabilities.
These results establish OpenART as a scalable foundation for studying agent safety in complex, evolving environments.