Understanding how AI agents behave in complex, long-horizon tasks is a growing challenge. Traditional methods rely on pre-built classifiers or manual annotation, which often miss novel patterns. Researchers from Hugging Face propose a new approach inspired by grounded theory, a qualitative method from the social sciences used for decades to systematically derive theories from data.
Their new system, AutoTraceGT (Automated Trace analysis through Grounded Theory), is the first multi-agent pipeline to automate grounded theory coding on agent trajectories. It iteratively performs open, axial, and theoretical coding until reaching saturation, producing a behavioral taxonomy tailored to each task. The method includes a principled saturation criterion and an auditable trail from raw data to final theory.
In experiments across six trajectory corpora, AutoTraceGT-generated codebooks recovered 73-91% of the failure modes present in human-annotated taxonomies. Moreover, the system surfaced additional behavioral patterns that human taxonomies missed, and the emergent theoretical narrative aligned with prior expert accounts.
When used as a deductive feature space, the codebook outperformed zero-shot and few-shot LLM baselines on downstream failure prediction tasks. The authors argue that grounded theory offers a scalable analytic tool for ML researchers and agent developers seeking to understand what agents actually do, rather than what they are expected to do.