A newly posted Hugging Face paper, From Evidence to Action: How Tool-Using Agents Fail, examines a gap that standard success metrics can hide: tool-using agents may reach the right end state without having established the evidence that justified their actions.
To study this, the authors introduce SafeActBench, a benchmark of 656 cases spanning six operational domains and five protocols. The protocols progress from static action judgment and investigated non-action to single-action and dependency-constrained multi-action workflows. A provenance-bound Evidence Ledger and a deterministic trajectory evaluator, which does not rely on an LLM judge, track what information was established, when actions occurred, and whether downstream dependencies were satisfied.
Across ten model-harness configurations, the paper reports that strong static action assessment can coexist with much weaker interactive execution. For three configurations re-evaluated on the same V1 cases, static accuracy was ≥95% while interactive success was ≤52%.
Failures often begin before execution. Agents stopped before completing the required investigation in 21.7–62.9% of V0 episodes, and 37.0–66.9% of V1 action attempts occurred before the required evidence was established. Once required evidence was obtained, single-action execution was usually reliable, while multi-action workflows additionally exposed unresolved prerequisites and incomplete execution.
Controlled interventions on 43 V1 cases across three configurations further showed that missing evidence does not necessarily stop agents: after one decisive record was withheld, agents still acted in 46.5–53.5% of completed episodes. The authors conclude that failures arise not only from missing information, but also from how agents use established evidence when deciding and executing actions.