Coding agents depend on a harness — the surrounding system that handles planning, tool use, and context management — yet most evaluations treat the harness as a single monolithic block. A new empirical study from Hugging Face argues this makes it impossible to tell which components actually contribute to long-horizon software-engineering performance.
To address this, the researchers built a lightweight, modular coding harness with a fixed execution loop while varying three components: planning, action space, and context management. They evaluated 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space, testing four models on SWE-Bench Verified and Terminal-Bench 2.1.
The findings are nuanced. Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures that would otherwise cut execution short. Among strategies, staging rule-based elision before LLM-based summarization delivered the strongest overall efficiency, while making elided content recoverable added machinery models rarely used and produced no accuracy gain.
Planning behaved differently depending on model strength: for weaker models it acted as an accuracy scaffold, keeping trajectories alive long enough to attempt an edit at extra cost; for stronger models it mainly removed redundant post-edit verification, lowering cost with little accuracy change. Meanwhile, predefined tools helped models with weaker bash proficiency, whereas bash-capable models performed effectively with a bash-only interface at substantially lower cost, especially on command-line-centric tasks.
Trajectory-level analysis explains the effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. The authors say the results point toward model- and budget-aware harness design and offer a modular framework for evaluating future harness components.