2026-09-03

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling — arXiv 2608.26623 (score 1.75)

• TL;DR: 3,808 instances over six workflow-DAG topologies and three difficulty tiers, six judges from 20B to frontier scale, paired with and without ground truth. On hard queries without ground truth, all six converge into a 77–82% alignment band regardless of scale.
• Ground-truth exposure reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp). Chain-of-thought and temperature are negligible; rubrics give up to +6.5 pp but do not generalise across judge–generator pairs.
• Why read it: it puts a hard ceiling on a harness component everyone treats as a fixed point. Where agent results are judged by an LLM on hard tasks, the reported gap between two systems can be smaller than the judge's own error.
• → AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement — arXiv 2609.01481 (score 1.75)

• TL;DR: a planning–coding–testing loop layered above three existing harnesses — Codex/GPT-5.5, OpenCode/DeepSeek-V4-Pro, Pi/MiniMax-M3 — improves all three. +52.25% average relative, max 82.86% after three iterations, plus a 70+ iteration multi-day build.
• No absolute score is published, so the relative gain cannot be sited against any other system; the game demo has no baseline and no human evaluation.
• Why read it: models held fixed, only the layer above the harness varies, and all three improve — another measurement of how much reported agent capability lives in the scaffold. Read it against the pick above: nothing says how its three benchmarks score their outputs, so whether 52.25% sits above or inside AgentJudgeBench's band is undetermined.
• → Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers — arXiv 2609.01343 (score 1.49)

• TL;DR: looped-Transformer results are normally compared at fixed model size, which hands the looped model extra FLOPs. SMELT matches per-token FLOPs, non-embedding parameters and KV cache — and the advantage survives: 6.8–18.0% of training FLOPs saved on the compute-optimal frontier, fitted as a separate Chinchilla-style law across four sizes to 54B.
• Two findings cut against how scaling results get read: the downstream gain exceeds what validation loss predicts (largest on Code), and it grows with sample length and in-context examples.
• Why read it: it is the rare scaling paper that argues a prior frontier was measured against the wrong control and then reports the effect anyway once the control is fixed.
• → SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

• Moonshot seeks revenue share from US cloud giants — Trivium China, 2026-09-01. A business-model story at weight 0.7 and triviumchina.com is blocked here, so only the headline was readable. No page written. Worth watching because it is the first sign of a Chinese lab pricing distribution rather than inference.
• The Alignment Science index could not be read, and that is recorded as unverified rather than as "nothing published." alignment.anthropic.com answers EGRESS_BLOCKED; a targeted search surfaced no post newer than the four already captured. This is the source whose two missed posts ran +73 and +38 days late, so an unverifiable check on it is itself the signal.
• eval-snapshots.yml has not fired on its own schedule for eight days. Sixth consecutive hand dispatch. Nothing has been lost because the dispatch works, but a scheduled check that only runs when someone remembers is not scheduled.

No new entity, concept or person page today — nothing here needs user review.

• Gemini 3.8 Flash (new — model page)
• Gemini 3.8 Flash Cyber (new — model page)
• AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling (new — paper page)
• Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement (new — paper page)
• SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers (new — paper page)

• Google DeepMind: two Recent Activity entries — the 3.8 Flash pair, and the Fairwind Program; both models added to Models & Products
• AI-Enabled Cyberattacks: new ## State of the Art (2026-09-03) opening with the three-lab gating comparison; one timeline row
• Eval Harness Configuration: new dated section — the judge ceiling, the 9.2-vs-1.2 split, and a version-hygiene rule for Terminal-Bench
• Agents (LLM Agents): two new State-of-the-Art entries and two Key Papers rows
• Qwen 3.8 Max and Alibaba / Qwen AI Lab: the 0902 snapshot, with the two Terminal-Bench generations separated
• Post-Training Scaling: SMELT added to Key Papers
• Gemini 3.7 Flash, Gemini 3.5 Flash Cyber: successor links