2026-08-23

1. The benchmark is being optimized against — and the seed moves the score more than the method does

• Hugging Face's Measuring benchmark optimization in speech recognition (2026-08-21) introduces three tests for "benchmaxxing" and applies them to 11 widely used open-source ASR models. Several of the highest-scoring systems reproduce the reference transcripts of VoxPopuli English and LibriSpeech when the audio does not support them: producing words absent from the audio, recovering silenced numbers at elevated rates, and — where the audio equally supports two written forms — picking the variant that particular benchmark expects (source)
• That third mode is the one to keep: it is not a memorised answer but a memorised transcription convention, which no amount of held-out audio would catch. The stated remedy is fully held-out sets and not reading WER off a single public benchmark
• Separately, Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See (arXiv:2608.17744) reports as its first result that changing only the random seed moves the accuracy score 7.7 points at 3.6–4.0B active parameters — more than every data and recipe effect it measured
• Why it matters: Eval Harness Configuration now holds four independent sources of variation — harness (6.8 points), context quality, memory, and now the seed at 7.7 — each comparable to the deltas release notes are written about. A single-benchmark delta below that floor carries no information, and a delta on a public benchmark carries less than the same delta on a held-out one
• → Eval Harness Configuration, Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See (arXiv:2608.17744)

2. GLM-5.3 gets its first independent number, and it is smaller than the vendor's claim

• Today's Sunday capture lists GLM-5.3 (max) at Artificial Analysis Intelligence Index 60, $0.68 per task, 1M context — the model was absent from the 2026-08-16 capture (source)
• Against the same table's GLM-5.2 (max) at 53, that is a 7-point move on a third-party composite between two models Z.ai states share an unchanged base — the first support for Post-Training Scaling's central claim from a party with nothing to sell
• It is not the same claim. An index composite is not the ~50% coding improvement of Z.ai's internal evaluations, and (max) compares each model's top reasoning setting. The honest reading is one tier, not a doubling. The weights are still not out — the stated safety-evaluation window closes around 2026-08-28
• Why it matters: "capability stopped arriving in the weights" has been a reading of release notes since W33. This is the first time an independent measurement has moved in the direction the claim predicts — and the first time anyone has bounded how far
• → GLM-5.3, Post-Training Scaling, Z.ai

3. Two papers say task success is the wrong number, from opposite ends of the embodied stack

• SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation (arXiv:2608.18701) pairs policy-visible tactile observation with evaluator-only finite-element ground truth and finds that across Diffusion Policy, π₀.₅ and FastWAM, all 12 in-distribution configurations contain successful rollouts that violate the deformation tolerance — 0.7–24% of each configuration's successes. A policy passes every existing benchmark while crushing the object
• Its second finding is the one that should change designs: touch raises success in all six out-of-distribution comparisons but DSR in only five, and in-distribution the benefit is mixed — "making touch available does not by itself ensure effective multimodal fusion"
• Decision-Metric Alignment in Latent World Models: Diagnostics and Action-Conditioned Objectives for MPC Planning (arXiv:2608.18746) names what a JEPA-style planner actually needs — does latent goal-distance rank plans by real progress — and shows it does not follow from representation quality: DA-LeWM improves online success while probe scores remain similar
• Why it matters: both replace a convenient proxy with an independent channel the system under test cannot see. That is the same correction the ASR post makes in story 1, arriving in physical control on the same day
• → Embodied Agents, Mechanistic Interpretability

4. A question this wiki left open, naming today as its resolution date, resolved as written

• On 2026-08-19 Qwen 3.8 27B recorded a third-party Intelligence Index 52 from a named author, with the note that it was not a held measurement and that "the next capture is 2026-08-23"
• Today's capture carries three Qwen3.8 27B rows where 08-16 carried none: xhigh 52, medium 44, low 43, all at 256k (source)
• The spread is the new information. 43 → 52 is 9 points on the same weights, larger than the gap to either model the original claim called it "neck and neck" with. Grok 4.6 gained three settings in the same capture with the same 9-point spread — and its high (61) scores above its xhigh (60) at four fifths of the cost, recorded as read
• Why it matters: a leaderboard row names a configuration, not a model. Quoted without its setting it names nothing — and this wiki has been citing such rows across Grok 4.6, GLM-5.3 and Qwen 3.8 27B all month
• → Qwen 3.8 27B, Grok 4.6