Evaluating large language model (LLM) agents is crucial for their development, but it has become prohibitively expensive. A single evaluation pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, and this cost is repeated across iterative development cycles. Existing methods like benchmark distillation reduce the number of tasks but do not address the cost of executing each task.
In a new paper, Hugging Face researchers propose early outcome prediction, a complementary approach that cuts costs within each task. The key insight is that an agent's final outcome is often evident from its intermediate behavior long before the run completes. Based on this, they introduce EarlyEval, a lightweight framework that trains LightGBM classifiers to predict success or failure from behavioral, textual, and reference-solution features.
EarlyEval halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. This allows for significant cost savings without substantially affecting evaluation accuracy.
Across three benchmarks—SWE-bench Verified, TerminalBench, and Toolathlon—EarlyEval eliminates 13% to 26% of agent steps and up to 44.1% of input tokens and 29.4% of output tokens, while maintaining 89% to 97% prediction accuracy. The per-agent resolve rates are perturbed by only one to two percentage points on average, demonstrating that early stopping can be both efficient and reliable.