As large language models (LLMs) are increasingly deployed within agentic systems, their performance depends not only on the model weights but also on the surrounding harness—the prompts, tools, control flow, memory, and orchestration code. This makes automated harness optimization, the iterative improvement of a harness guided by evaluation, a crucial capability for AI systems. Yet, until now, there has been no common protocol to measure how well frontier LLMs perform at this task.
To address this gap, researchers have introduced HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. In this setup, an optimizer (an LLM paired with a coding harness) receives a target agent's seed harness, graded evaluation feedback, and a fixed evaluation budget. The optimizer edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout the search.
A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. This design ensures that the benchmark measures genuine improvement rather than overfitting to the evaluation set.
The study evaluated five frontier LLMs as optimizers, both under a shared coding harness and under their native harnesses, across four downstream tasks and over 111 scored runs. The results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes.
These findings establish harness optimization as a measurable and discriminative capability with significant room for improvement, offering a valuable direction for evaluating how well LLMs can actually improve agent systems.