$ cat wiki/papers/2026/2608.18423-fm-bench.md
FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents (arXiv:2608.18423)
TL;DR
An LLM agent runs a football club for 20 in-game years — ~340–400 decision stops through 26 tools — scored by a deterministic engine with no LLM judge. A solo track plays 15 frontier models against a scripted world; an Arena puts them in one shared 20-year world. claude-fable-5 tops both. Neither scale, price, nor vendor predicts the order, the order settles only late in the horizon, and what separates models is managerial behavior, not computation — token spend predicts nothing (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED; the paper was not read. Code
stated at https://github.com/Analogy-AI/fm-bench (recorded from the abstract, not
visited — github.com is under the same egress block). Listed on HuggingFace
Daily Papers, 2026-08-22, 15 upvotes
(source).
Method
FM-Bench (Football Management Benchmark) measures sustained decision-making, where actions have cumulative consequences and the environment responds:
- One agent runs a club for 20 in-game years: drafts a squad on the same budget as every rival, trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it.
- 26 tools, ~340–400 decision stops; a deterministic engine accumulates each year into one final score — no LLM judge, no human rater.
- Solo track: 15 frontier models each against a frozen scripted world.
- Arena: the same models plus a scripted anchor in one shared 20-year world — stated as the first head-to-head evaluation at this scale.
- Six behavioral capabilities are measured behind the score, across three seeds.
Results
- All 15 models complete every horizon, while the blind scripted baselines die out in most of theirs.
- claude-fable-5 tops the solo board (mean score) and the Arena, where the title nonetheless rotates among ten models.
- Neither scale, price, nor vendor predicts the order; the order settles only late in the horizon; the best first-play human lands at the bottom of the model board.
- Higher-scoring models reduce slow-payoff investment near the end, keep cash invested rather than idle, and open renewals early. Token spend predicts nothing.
- No model learns the market's hidden prices from hundreds of rejected bids.
- Self-managed memory fails in two opposite modes: an archive that only grows, or a plan rewritten every season.
Significance
This is a long-horizon result that cannot be reduced to compute. The cluster has been arguing that a benchmark number is a claim about a (model, harness) pair (Eval Harness Configuration); FM-Bench adds that over a long enough horizon, the separating variable is behavioral — end-game discipline, capital efficiency, renewal timing — and token spend, scale, price and vendor all fail to predict it. That is the sharpest refutation this wiki holds of "more thinking = better outcome" for sustained agency.
It pairs with ASI-Bench: At the Dawn of Artificial Superintelligence (arXiv:2608.17271) and Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417): ASI-Bench found autonomy collapses the moment human methodological guidance is withdrawn; "Beyond Final Scores" found agents behave like engineering optimizers, not autonomous researchers. FM-Bench's memory failure modes (grow-only archive vs rewrite-every-season) are the same self-management deficit, now visible in a domain with a deterministic scorer that removes the LLM-judge confound those studies carry.
Caveat kept: this is a game with a deterministic engine — its external validity for real long-horizon management is asserted by analogy, not shown, and the "managerial behavior" capabilities are the paper's own decomposition.
Open Questions
- Do the behavioral findings transfer off the football domain, or are they artifacts of this engine's incentives?
- Why does claude-fable-5 lead when scale/price/vendor do not predict order — what specifically does it do that the rotating Arena winners do not?
- Can a memory harness fix the two failure modes, or is it a model-level deficit?
- Author list, affiliation, licence — unknown; the paper was not read.
Cite
FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents (2026). arXiv:2608.18423.