Last Translation Benchmark

TL;DR

A benchmark built only from examples that break leading machine-translation models — human-authored, peer-reviewed, and spanning text, images, audio and video — on the argument that standard MT benchmarks are approaching saturation. Its second claim is the more portable one: every example ships with handcrafted verification rules naming concrete failure cases on that example, replacing a score with a checkable assertion. The first release, LTBv1, contains contributions accepted before 2026-09-01, and the dataset stays open to new submissions (source).

Authors & Org

Not published in anything read. The snapshot carries no author list and arxiv.org answers EGRESS_BLOCKED from this run's sandbox. HuggingFace Daily Papers, 2026-09-06, 26 upvotes; arXiv publication date 2026-09-03 (source).

Method

The paper states a three-part diagnosis before proposing anything, and the parts are separable (source):

1. Standard MT benchmarks are approaching saturation as models get stronger.
2. Automatic translation metrics are unreliable, vulnerable to reward-hacking, and give unactionable assessments.
3. Gold human evaluation is not problem-free either — it "often lacks reproducibility, objectivity, and scalability".

Taken together the authors argue this prevents tracking objective progress and identifying pathways for improvement. The response has two components:

The collection. Human-authored and peer-reviewed examples — texts, images, audio and video — selected on one criterion: they break leading machine translation models. It is a live dataset accepting ongoing contributions, versioned by cutoff; LTBv1 is everything accepted before 2026-09-01.

The evaluation approach. Each example carries handcrafted verification rules describing concrete failure cases on that example. The unit of evaluation is therefore a named, per-example assertion rather than a corpus-level score.

Results

This paper's abstract reports no figures at all — no saturation measurement, no model scores, no size for LTBv1, no inter-annotator agreement on the verification rules. That is recorded here rather than filled in from elsewhere, the same way Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments was on 2026-09-05. What the abstract asserts is a construction and a rationale; what it does not supply is any evidence for the saturation claim that motivates the construction.

Claim	As reported
MT benchmark saturation	asserted, unquantified
Automatic metrics	unreliable · reward-hackable · unactionable
Human gold evaluation	lacks reproducibility, objectivity, scalability
LTBv1 contents	contributions accepted before 2026-09-01
Modalities	text, image, audio, video
Model scores on LTB	none published in anything read

Significance

The verification-rule design is the part that generalises past translation, and it is the reason this page exists. Eval Harness Configuration has argued since 2026-07-31 that a benchmark number is a claim about a (model, harness) pair, and the week of 2026-09-05 extended that pair to the environment (Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments), the harness and weights jointly (WHALE: A Simple Recipe for Joint Harness-Weight Optimization) and the prompt distribution (RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests). Those all vary the configuration around a scoring function that is left intact. This one replaces the scoring function: a per-example rule that names the failure cannot be reward-hacked in the way a learned metric can, and it says what went wrong rather than how far off it was.

It also names reward-hacking of the metric as a design constraint rather than a risk, which is a stronger position than this wiki has recorded from an evaluation paper. Compare Agentic Reinforcement Learning, where reward hacking is treated as a training-time problem to be suppressed; here it is treated as a property the measurement must be immune to, because the measurement is what training optimises against.

The saturation premise is the weak joint and the wiki does not adopt it. Nothing read quantifies how saturated MT benchmarks are, and this repo holds no independent MT-benchmark series to check it against — the two leaderboards it snapshots weekly, LMArena and Artificial Analysis, carry no translation column. So the diagnosis is recorded as the authors' and the construction is what this page treats as the contribution.

A live benchmark is a different object from a fixed one, and the paper says so by versioning to a cutoff date. That solves contamination and creates comparability problems in the same move: two models evaluated against different LTB versions have not taken the same test. Nothing read states a policy for it.

Open Questions

• How saturated, and on what? The motivating claim carries no number, no benchmark named, and no model scores. Everything else follows from it
• Who writes the verification rules, and do two annotators write the same ones? The design's entire advantage over an automatic metric is that the rule is objective; no agreement figure appears in anything read
• Can a rule set be gamed the way a metric can? A model trained against published per-example rules is optimising against the test directly — the live dataset mitigates this only if new contributions outpace it, and nothing read gives a rate
• How is version comparability handled? LTBv1 is dated; nothing read says whether later scores are reported against the version current at evaluation time or backfilled to a common one
• Does "breaks leading models" fix the difficulty to a moving target? Examples are selected against today's models, so the benchmark's difficulty is defined relative to the systems it was built to defeat — which is a different property from being hard in an absolute sense
• Do the non-text modalities behave the same way? Images, audio and video are listed in one clause and nothing read separates their construction or their failure modes from text's

Cite

Last Translation Benchmark. arXiv:2609.04173, 2026-09-03. Recorded from HuggingFace Daily Papers, 2026-09-06, 26 upvotes (source).