Researchers have introduced ActiveSaddler, a method that reframes a largely overlooked part of LLM agent harness optimization: deciding which training scenarios should drive the feedback used to improve an agent's prompts, tool interfaces, and control logic.
Existing harness optimizers, the paper argues, focus on how the harness is updated while keeping the scenario order fixed before optimization begins. But as a harness evolves, the scenarios that are most useful for further improvement can change — meaning the training curriculum itself should adapt alongside it.
ActiveSaddler models this evolving curriculum as a non-stationary bandit with dynamically instantiated optimization targets. It abstracts recurring failures into reusable "failure-pattern arms," estimates the potential learning progress from targeting each pattern, and balances revisiting known weaknesses against exploring unseen scenarios to discover new ones.
Optimization outcomes continually update both the set of discovered failure patterns and their priorities, allowing the curriculum to co-evolve with the harness. Experiments on GAIA2 and Terminal-Bench 2.0 show ActiveSaddler consistently discovers stronger harnesses, improving test Pass@1 by 4.4 and 7.5 percentage points over the same harness optimizer using a scenario order fixed before optimization.
Ablations indicate the gains depend on dynamically constructing optimization targets, estimating their evolving utility, and balancing continued optimization with new failure discovery. The authors position automated curriculum learning as a new, crucial optimization dimension for harness optimization. Code is available in the linked repository.