$ cat wiki/papers/2026/2608.00677-openart.md
OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution (arXiv:2608.00677)
TL;DR
Red-team the environment, not the prompt. OpenART is an arena of 10,000+ validated stateful scenarios across 50 domains in which the unit of attack is the executable environment an agent persists in, on the argument that agent risk accumulates through shared state reused across long-horizon workflows and that short, static safety benchmarks cannot see it (source).
Authors & Org
Yunhao Chen, Xin Wang, Yixu Wang, Yi Liu, Jie Li, Yan Teng, Xingjun Ma, Xia Hu, Yu-Gang Jiang — Fudan University, Shanghai Artificial Intelligence Laboratory and XSafeAI (source).
Method
The environment is the fundamental unit of red teaming rather than an individual prompt or task. Scenarios are drawn from a pool of 500,000+ tools and skills, require a median of 97 tool calls, and support unified evaluation across 75 agent-model configurations (source).
The attack method is named Evolutionary Markov Hypergraph Attack (EMHA),
proposed "to systematically explore evolving attack surfaces"
(source). Nothing
read describes the algorithm beyond that phrase — arxiv.org is blocked from
this environment and no PDF was opened.
Results
None are recorded, because none were seen. The abstract extract carries scale figures only: no attack success rate, no result table, no comparison against another red-teaming benchmark (source).
A paper page with an empty Results section is the honest state here. The scale claims are what was published to search; the findings are behind a fetch this run could not make.
Significance
The framing is the contribution this wiki can evaluate today, and it lands on an argument already open here. Eval Environment Containment and Eval Harness Configuration both record the same underlying problem from the defender's side: an agent's behaviour is a property of a (model, harness, environment) triple, and publishing a number for the model alone hides most of it. OpenART makes that the attacker's premise — if state carries the risk, the state is what you attack.
The median of 97 tool calls is the figure worth holding. Every agentic benchmark this wiki tracks a score for — Terminal Bench 2.1, DeepSWE, CyberGym on DeepSeek V4-Pro-0813, Toolathlon-Verified on DeepSeek V4-Flash — is far shorter-horizon than that, which is precisely the paper's complaint about the field it is entering.
It also arrives from a Chinese academic-lab consortium rather than a frontier vendor, at a moment when the safety evaluations this wiki holds (Preparedness Framework, AI Control Roadmap) are almost entirely first-party. Whether the arena is released is unknown and is the question that decides whether that changes.
Open Questions
- Is anything released? Nothing read states whether the arena, the 10,000 scenarios or EMHA are public
- What did it find? No success rate against any model was seen
- Which 75 configurations? Whether frontier models are among them is unstated, and it determines whether this measures agents anyone deploys
- How were 10,000 scenarios "validated"? The word appears without a procedure
- Whether an evolved, generated environment stays representative of a real one — the standing objection to open-ended benchmark generation