AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

TL;DR

LLM judges are used to score agentic tool-calling systems, and nobody had checked whether they can. AgentJudgeBench does: 3,808 instances, six workflow-DAG topologies, three difficulty tiers, six judges from 20B to frontier scale. Judge alignment degrades monotonically with task difficulty, and on hard queries without ground truth all six judges converge into a 77–82% band regardless of scale — a ceiling that model capacity does not lift (source).

Authors & Org

Not published in anything read. The HuggingFace Daily Papers snapshot carries title, abstract and upvote count but no author list, and arxiv.org answers EGRESS_BLOCKED from this run's sandbox, so no authors or affiliation are recorded rather than guessed. HuggingFace Daily Papers, 2026-09-03, 16 upvotes; arXiv publication date 2026-08-27 (source).

Method

Scope is deliberately narrower than "LLM-as-a-judge": this is judging agentic tool-calling over workflow DAGs, as distinct from judging open-ended text or preference pairs (source).

| | |---|--- | Instances | 3,808 | DAG topologies | 6 | Difficulty tiers | 3 | Generators | 5 — open-weight 3B–70B, plus GPT-5.4 | Judges | 6, from 20B to frontier scale | Conditions | paired, with and without ground truth The with/without-ground-truth pairing is the design choice that produces the paper's sharpest results, because it separates "the judge is weak" from "the judge has nothing to check against".

Results

The ceiling. On hard queries without ground truth, all six judges land in 77–82% alignment irrespective of scale. The paper attributes the ceiling primarily to task difficulty, and notes its height is partly prompt-dependent for weaker generators (source).

Degradation is 1.5× faster without ground truth.

Ground truth is not uniformly helpful. Exposure to it reduces alignment for GPT-5.4 by 1.5 pp and Gemini-2.5-Pro by 3.9 pp, which the paper reads as over-anchoring. This is the finding that should change practice: the intuition that handing a judge the reference answer can only help is wrong for two of the models tested.

Mitigations, ranked by whether they work:

Intervention	Effect
Chain-of-thought reasoning	negligible
Judge temperature	negligible
Structured evaluation rubrics	up to +6.5 pp, but does not generalise uniformly across judge–generator pairs
Best judge depends on the yardstick. With ground truth, QwQ-32B best
matches the programmatic reference; a human validation study identifies
GPT-OSS-120B as the most human-aligned. Without ground truth, frontier judges
lead "only marginally within the shared ceiling"
(source).

Significance

A 32B model beating frontier judges against a programmatic reference, and a different 120B model winning the human study, means "which judge is best" has no answer independent of what the judge is being scored against. That is not a tuning detail — most agentic evaluation in circulation reports a single judge and a single number.

It lands directly on two threads this wiki keeps:

• Eval Harness Configuration — the accumulating argument that the harness is a variable and often the dominant one. A judge is part of the harness, and this paper puts a hard ceiling on that component specifically.
• J-Zero: Unified Challenger–Solver–Judge Co-Evolution from Zero Data, captured 2026-09-01, whose whole design move is to avoid scoring content with a judge — deriving the Judge's training signal from how a response was produced instead. AgentJudgeBench measures the failure mode J-Zero routes around, from the other direction and without reference to it. Neither cites the other; the pairing is this wiki's.

The narrow reading also matters for Agents (LLM Agents): if agentic tool-calling results are being graded by judges that plateau at 77–82% on the hard cases, the reported spread between two agent systems can be smaller than the judge's own error.

Open Questions

• The 77–82% band is measured across six judges up to frontier scale. Whether it is a property of the task or of the judging format is not separated — and the rubric result (+6.5 pp) is weak evidence that format matters somewhat.
• Rubrics help but "do not generalize uniformly across judge–generator pairs", which leaves practitioners with a per-pair tuning problem and no rule.
• Nothing read reports how the programmatic reference itself was validated, which is the anchor the QwQ-32B result depends on.

Cite

arXiv 2608.26623, AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling, published 2026-08-27 — https://arxiv.org/abs/2608.26623 → (snapshot)