As software production increasingly passes into the hands of software itself, coding agents are being commissioned to build other agents tailored to users' own workflows. This delegation rests on a quiet assumption: that the delivered agent will faithfully serve the requirement it was built for, rather than merely appear to. According to the paper's authors, no existing benchmark tests that assumption.
To fill that gap, the researchers introduce AgentSWE, a benchmark for agent software engineering that spans three stages of the agent development lifecycle: Creation of a complete agent from a natural-language requirement, Editing of production agent codebases, and Optimization toward a measurable target.
Each of the benchmark's 25 tasks is framed as a commission rather than an exam. The builder iterates against a handful of illustrative cases, while acceptance exercises the frozen submission on held-out cases designed to span the full requirement. Grading is done against requirement-derived rubrics under hard pass/fail gates or, for Optimization, against the source benchmark's own objective.
Evaluating a range of coding agents across models and harnesses, the authors report that none comes close to delivering what a requirement asks for. The gap between what a builder demonstrates and what it delivers is described as the norm, not the corner case.
An accompanying empirical analysis of failure trajectories identifies recurring failure modes, including overfitting to the visible cases, hollow runtimes behind code that looks complete, and unsupported claims of completion. The authors frame each of these as a signal by which coding agents can learn to build the agents they are asked for.