As AI agents move from research prototypes to deployed tools, their performance increasingly depends on the external execution infrastructure, or "agent harness," that surrounds the model. Changing this harness while keeping model weights fixed can substantially alter task outcomes. Yet most evaluations report performance under a chosen harness, leaving the model's ability to develop the harness itself largely unexplored.
To address this gap, researchers introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. It comprises two stages: Creation, where an agent starts from a minimal seed and a small number of cases to build a complete execution system, and Evolution, where the agent iteratively revises its own created harness using downstream execution feedback to improve benchmark performance.
The study evaluates harnesses on capability (task success on held-out benchmarks) and efficiency (execution-token cost). Results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique instances, with hidden evaluation tasks withheld from development. Generated harnesses remain substantially behind mature human-engineered references on code and search/research, but match or exceed references on writing and machine-learning experimentation, with large variation in execution cost.
Evolution yields some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model show that gains depend strongly on the model executing the harness, indicating limited transfer across models. The authors highlight that this work targets the engineering-heavy layer that everyone relies on but rarely evaluates.