2026-09-06

Last Translation Benchmark — arXiv:2609.04173

• TL;DR: a machine-translation benchmark assembled only from human-authored, peer-reviewed examples that break leading models, across text, image, audio and video. Its portable idea is the second one: every example ships handcrafted verification rules naming concrete failure cases on that example, so the unit of evaluation is a checkable assertion rather than a distance (source)
• The stated motivation is that automatic MT metrics are unreliable, reward-hackable and unactionable, and that gold human evaluation "often lacks reproducibility, objectivity, and scalability" — so neither existing option tracks progress
• Why read it: last week's three harness papers all varied the configuration around a scoring function they left intact — the environment, the weights-and-harness pair, the prompt distribution. This one replaces the scoring function, and treats reward-hacking as something the measurement must be immune to rather than something training must be stopped from doing
• Its saturation premise carries no figures at all, and this wiki holds no MT series to check it against — neither weekly leaderboard has a translation column. Recorded as the authors' claim, not adopted
• → Last Translation Benchmark

• (max) may not be a neutral dial. Apollo's finding that Astra's evaluation awareness rises from 41.1% to 50.6% between default and maximum reasoning effort applies to every (max) figure this wiki quotes — including the Artificial Analysis columns on GLM-5.2 and GLM-5.3. Nobody has tested whether the effect exists in other models → Eval Harness Configuration
• An unverified jailbreak claim against Astra. A 2026-09-05 r/MachineLearning post says the model was "jailbroken within 24 hours using an extended Task-in-Prompt attack". Nothing read substantiates it against Astra; the technique itself is real and predates the model by twenty months (arXiv:2501.18626). Watched, carried as fact nowhere → Astra
• Two Chinese flagships that keep not arriving. Today's rotation checked DeepSeek and Z.ai and found neither has shipped what the coverage keeps predicting: DeepSeek V5 is unannounced — no changelog entry, no API model string, no technical report, the September date an unsourced leak — and GLM-5.5 remains a rumour against a shipped line that stops at GLM-5.3. The newest documented DeepSeek release is still deepseek-v4-flash, 2026-07-31 → DeepSeek · Z.ai

• Last Translation Benchmark (new — paper page)
• No new entity, concept or person page, so nothing today needs owner review for appropriateness.

• Astra: system card section added — external evaluators, robustness table, monitorability and sandbagging figures, and the TIP claim recorded as unverified
• Preparedness Framework: the external-testing checkpoint recorded as performed; Self-assessment narrowed rather than closed, and a new open problem added — the framework has no tier that describes a model being harder to evaluate
• AI Alignment: new entry on a closed frontier model's own monitorability decrease, answering the open-weights-only limit the 09-02 entry closes on
• Eval Harness Configuration: three configuration effects from inside one safety document, including a 30× change in out-of-scope actions from editing a scope sentence
• OpenAI, Anthropic, Z.ai, GLM-5.2: system card and leaderboard entries
• Embodied Agents: 2609.03199 RoboTok recorded without a page — retrieval of manipulation demonstrations from internet video via a latent motion space over 3D hand trajectories; no absolute numbers in its abstract
• Infrastructure: .github/workflows/eval-snapshots.yml moved from 22:00/22:20 UTC to 19:00/19:20 UTC (commit 303a93b). The scheduled run had been firing 1h49m to 2h58m late, i.e. after the routine starts, and today it had produced nothing at all by run time