A new research paper posted to Hugging Face introduces the Agent Error Dataset (AED), a large-scale resource built on the premise that a failed LLM agent rollout carries more useful information than its final reward signal alone. The dataset contains 50,228 error-diagnosis pairs sourced from 9,961 tasks spanning 33 environments, 19 harness families, and 23 policy models in text-based agent systems.
The authors argue that reusing failed experience for learning requires pinpointing a decision to revise and testing a concrete alternative. To that end, they retain source traces and execution metadata, enabling cross-setting failure analysis and re-diagnosis without repeating the original rollout.
At the core of the work is a five-stage Agentic Error-to-Training (AET) pipeline. It collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, corrections are compared with original-action retries from the same checkpoint under matched execution settings. Separate training views are then constructed for diagnosis and actor recovery.
The reported results are notable. Across 3,062 matched replay pairs, first-proposal corrections raised verifier pass rates from 18.4% to 51.1% — a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks lifted Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in the comparison scored 54.7%, and mean agreement improved at each of four increasing training-set sizes.
In a single-seed comparison of actor-training recipes, action-only repair training scored 6.67 percentage points higher on WebShop-lite than success-only training. The paper positions AED and the AET pipeline as tools for scaling failure analysis and error-aware post-training in agentic systems.