Researchers behind a new Hugging Face paper, TraceDance, propose an automated system that converts real-world agent deployment traces into targeted benchmarks for behaviors developers actually encounter in production, rather than relying only on fixed benchmark suites.
The system targets user-specified undesirable behaviors. Its Anchor-and-Confirm method pairs programmable retrieval with candidate-level confirmation by a Flash large language model, while an Anchor Synthesis Loop generates and revises specifications for custom behaviors.
Evaluation uses decision-point continuation: an LLM's next turn is assessed at a recorded decision point against a behavior-specific rubric, with no reference answer and no environment replay required.
Experiments in coding and general tool use drew on 252,557 sessions, yielding 107 benchmarks with 4,125 instances and fulfilling 95.3% of build-target requests. Human annotators confirmed the requested behavior in 84% of sampled instances, and the automated grader's agreement with human pass/fail judgments was comparable to agreement between the annotators themselves.
Nine frontier LLMs achieved a mean pass rate of only 26.7%, indicating they still struggle to respond appropriately at the evaluated decision points. The authors suggest TraceDance could become a key component of a recursive self-improvement loop by turning deployment problems into targeted benchmarks.