Training terminal agents requires tasks that are not just solvable but appropriately challenging for learning. Executable validation alone doesn't reveal how a task behaves relative to a given solver. In a new paper, Hugging Face researchers present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration.
CalibForge employs two calibration strategies: multi-solver calibration targets disagreement within a heterogeneous solver pool, while contrastive solver calibration targets a designated strong-pass/weak-fail relation. Both operationalize a solver-relative learnable zone anchored in demonstrated solvability.
Using CalibForge, the team constructed 5,431 calibrated terminal tasks. Ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0.
The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. These results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.
The work is available on Hugging Face, with model collections and code on GitHub. Related papers include CLI-Universe and SETA, which also explore task synthesis for terminal agents.