$ cat wiki/papers/2026/2608.16033-r3-bench.md
R³-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets (arXiv:2608.16033)
TL;DR
Gives a model six problems and one budget between them, and measures how well it allocates. An offline oracle built from the same model's demonstrated single-problem competence matches or exceeds the model's own contest performance in all 72 main-table cells and is strictly higher in 71. The gap is not capability — it is allocation (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only
(source).
Listed on HuggingFace Daily Papers, 2026-08-19, 11 upvotes — that community's popularity signal and nothing more (source).
Method
The frame is resource rationality from cognitive science: how should an agent allocate limited computation to maximise expected value. The stated defect in existing work is that reasoning and agent benchmarks use independent per-task budgets, and that prior shared-budget studies do not calibrate suite performance against the same model's demonstrated single-problem competence.
| Element | What it is |
|---|---|
| Suite | six problems under one shared budget |
| Domains | mathematics, competitive programming, abstract reasoning |
| Settings | tool-free and agentic |
| Oracle | matched single-problem response curves define an offline empirical oracle over observed successes |
| Diagnostics | trajectory analysis of strategy updating and failure patterns |
| The oracle is the design's load-bearing part: it is built from the model's own | |
| measured behaviour, so the comparison is a model against itself under a better | |
| allocation policy, not against a stronger model. |
Results
All figures as reported (source).
| Finding | As reported |
|---|---|
| Oracle vs contest, six models | oracle mean matches or exceeds contest mean in all 72 cells; strictly higher in 71 |
| Equal-allocation replay, moderate tool-free pressure | exceeds contest performance for four of six models |
| Fixed schedulers, strong agentic pressure | at least one fixed scheduler exceeds the contest mean in six of nine cells (three-model diagnostic) |
| Strategy updating | limited; failure patterns are pressure-dependent |
| Policy dominance | no policy dominates across domains |
| What the abstract does not give: the six models, the budget unit (tokens, | |
| calls, wall-clock), the size of any gap, the schedulers tested, or the benchmark | |
| composition. |
Significance
It is the fourth mechanism on Test-Time Compute (Inference-Time Compute Scaling), and it is the one that needs no new method to exploit. The three already recorded change what a budget buys: Gambit reallocates a live budget, the full-bandwidth transformer reduces what a step needs, and Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning (arXiv:2607.29211) argues the right budget is sometimes zero. R³-Bench changes none of those — it shows that models already underuse the budget they have, measured against their own demonstrated competence.
"Equal allocation beats the model's own choices for four of six models" is the sentence to keep. Splitting the budget six ways with no reasoning at all outperforms the model's reasoning about how to split it. That is a cleaner indictment than any of the three prior mechanisms, because the baseline it beats is free.
It converges with the same-day negative result from a different direction. How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) names the missing faculty as a metacognitive loop — checking your output, revising, questioning the path. Allocating a shared budget across six problems requires judging your own likely success on each before spending, which is the same faculty priced in tokens. Neither paper cites the other; the pairing is this wiki's and is labelled as such.
Consequence for how this wiki reads a benchmark number. Every reasoning score this repo records comes from independent per-task budgets. R³-Bench says that regime overstates what the same model does when it has to allocate — a condition closer to how an agent actually runs. Nothing here lets that be quantified as an offset, and no source read attempts one.
Open Questions
- How large is the gap? "Strictly higher in 71 of 72" is a direction, not a magnitude; a uniform 0.3-point edge and a uniform 15-point edge both satisfy it.
- What is the budget denominated in? Without a unit the result cannot be compared to the token-count claims on Test-Time Compute (Inference-Time Compute Scaling), which is the same incomparability defect Eval Harness Configuration tracks.
- Is an offline oracle over observed successes reachable in principle? It is built with hindsight over what did succeed; whether any online policy could approach it is the question the result implies and does not test.
- Which six models, and does the gap correlate with anything? Six models, three domains, two settings — no per-model result was read.
- Author list, affiliation, licence — unknown; the paper was not read.
Cite
R^3-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets
(2026). arXiv:2608.16033.
Related
- Test-Time Compute (Inference-Time Compute Scaling)
- Reasoning Models
- Eval Harness Configuration
- Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning (arXiv:2607.29211)
- How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905)