Using Grounded Theory for Agent Behavior Analysis at Scale

TL;DR

AutoTraceGT automates grounded theory — a six-decade-old qualitative method from the social sciences — over agent trajectories, iterating open, axial and theoretical coding until saturation to produce a behavioural taxonomy built for each task rather than chosen in advance. Across six trajectory corpora it recovers 73–91% of the failure modes in human-annotated taxonomies and surfaces additional patterns those taxonomies miss; used as a deductive feature space, its codebook beats zero-shot and few-shot LLM baselines on downstream failure prediction (source).

Authors & Org

Not published in anything read. The snapshot carries no author list or affiliation and arxiv.org answers EGRESS_BLOCKED from this run's sandbox. HuggingFace Daily Papers, 2026-09-08, 19 upvotes; arXiv publication date 2026-08-31 (source).

Method

The problem the paper names is that understanding agent behaviour needs methods that scale to thousands of trajectories and surface new patterns in long, often unfamiliar tasks where pre-built classifiers fall short (source).

Its answer is to import a qualitative method rather than invent one. AutoTraceGT is a multi-agent pipeline — described as the first to automate grounded theory on agent trajectories — that runs the method's three coding stages iteratively until saturation. Two properties are carried over from the original method and are the reason for choosing it: a principled saturation criterion (a stopping rule that is not a fixed budget) and an auditable trail from data to theory.

The output is a codebook, which the paper then uses two ways: descriptively, as a taxonomy of what the agents did, and deductively, as a feature space for predicting failure.

Results

Finding	As reported
Corpora	six trajectory corpora
Recovery of human-annotated failure modes	73–91%
Additional patterns	surfaces modes the human taxonomies miss
Narrative	emergent theoretical narrative aligns with prior expert accounts
Downstream failure prediction	codebook-as-features outperforms zero-shot and few-shot LLM baselines
The abstract carries no absolute numbers for the downstream prediction task —
no accuracy, no baseline figure, no per-corpus breakdown of the 73–91% range
(source).

Significance

This is the measurement cluster turning on its own instruments. Eval Harness Configuration spent August accumulating papers that vary the environment, the harness, the prompt distribution and the scoring function around agent evaluation. All of them still report a score. This one asks what the categories are before anything is counted, and finds that the hand-built taxonomies those scores are bucketed into are 9–27% incomplete.

"Pre-built classifiers fall short" is a claim about every failure-mode breakdown this wiki cites. Agents (LLM Agents) records repeatedly that an agent's own claim to have finished is worth almost nothing, and that succeeding once and succeeding reliably are forty points apart — findings that depend on somebody having named the failure modes correctly. A method that recovers most of a human taxonomy and finds more is evidence the naming step is a real source of error, not a formality.

The saturation criterion is what makes it operable rather than a nice idea. An LLM asked to "categorise these trajectories" will produce categories indefinitely; grounded theory supplies a stopping rule with a defence. That is the same property this wiki valued in Last Translation Benchmark's per-example verification rules — a procedure whose output can be argued with, rather than a number whose provenance is a prompt.

The auditable trail matters more here than the accuracy does. A codebook that can be traced back to the trajectories that produced it is reviewable by the people whose taxonomy it is contradicting. Nothing else in this cluster offers that.

It is also, quietly, an argument that the social sciences already solved this. The method is six decades old and unmodified; the contribution is the automation. This wiki holds no other instance of a frontier-agent evaluation problem being answered by adopting an existing qualitative methodology whole.

Open Questions

• What is the 73–91% spread explained by? Six corpora, an 18-point range, and no per-corpus figures in anything read — whether the low end is a hard domain or a bad fit is undetermined
• Are the "additional patterns" real failure modes or artefacts? The claim that the pipeline finds what humans missed is checked against nothing read; the human taxonomy is the only reference and it is the thing being contradicted
• Who validates saturation? The stopping rule is the method's strongest import, and nothing read states how saturation is detected by the pipeline or whether two runs saturate at the same codebook
• How much does it cost? A multi-agent pipeline over thousands of trajectories with iterative coding is an expensive object, and no compute, token or wall-clock figure appears in anything read
• Does the codebook transfer? The design point is a taxonomy tailored to each task; whether a codebook built on one corpus retains any value on another is the difference between a research tool and an evaluation standard, and is not addressed
• What are the downstream numbers? "Outperforms zero-shot and few-shot LLM baselines" carries no figure at all

Cite

Using Grounded Theory for Agent Behavior Analysis at Scale. arXiv:2608.30391, 2026-08-31. Recorded from HuggingFace Daily Papers, 2026-09-08, 19 upvotes (source).