Test-Time Compute (Inference-Time Compute Scaling)

Definition

Any technique that improves output quality by allocating additional compute at inference time. The trained model stays fixed, but it thinks/searches/verifies more during the process of producing an answer.

Representative forms:

• Chain-of-Thought (CoT) — explicitly generating intermediate steps
• Self-consistency — sampling multiple answers and taking a majority vote
• Tree-of-Thoughts / search — branching exploration
• Verifier-guided search — a separate verifier evaluates candidates
• Reflection / critic — self-criticism followed by retry
• Tool augmentation — calling external calculators/search

Why It Matters

• Matches personal interests (reasoning 1.3x)
• One of the biggest paradigm shifts of 2024-2026 — beyond "bigger models," toward "models that think longer"
• The core inference-cost trade-off, assuming training-cost limits have been reached
• The operating mechanism behind Reasoning Models

State of the Art (2026-07)

• OpenAI o-series successors (e.g., GPT-Rosalind) — domain-specialized reasoning
• Achieving Gold-Medal-Level Olympiad Reasoning via Simple and Unified Scaling — claims to achieve gold-medal olympiad reasoning through "simple and unified scaling" alone — a data point on how far test-time compute can be pushed
• Claude Opus 4.7 "multi-step tasks, thoroughness, consistency" — implicit test-time compute scaling

Two months on, the interesting question is no longer how much inference compute helps but where it is spent and whether it is the compute you think:

• Horizon, not parameters — Agents-A1 (2026-06) matches trillion-parameter models on agentic benchmarks with 35B active by scaling the agent horizon — trajectory length and ability breadth — rather than model size. Test-time compute spent on a longer trajectory bought what 30× the parameters did
• The compute may not reach the policy you trained — the MIPI paper (2026-06) finds RL optimising a training proxy while a different engine serves inference, so effort spent at inference is applied to a policy the training never targeted
• It became a price tier — Gemini 3.5 Pro puts "Deep Think" behind the $250/month Ultra tier. Where test-time compute was a research knob it is now a line item, which is its own answer to "how far is it efficient"
• Failure scales too — Ring-Zero documents behaviours appearing only above ~100B, including stalling on ambiguous context. More inference compute does not monotonically buy more reliability → Reasoning Models

2026-08-19 — a fourth mechanism, and it needs no new method at all

The three mechanisms below change what a budget buys. R³-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets (arXiv:2608.16033) changes nothing and shows that models already underuse the budget they have (source).

Its design is what gives it force. Six problems share one budget across mathematics, competitive programming and abstract reasoning, in tool-free and agentic settings. The comparison is not against a stronger model — it is against an offline oracle built from the same model's measured single-problem response curves. So the model is being compared to itself under a better allocation policy.

Finding	As reported
Oracle vs contest, six models	oracle mean matches or exceeds contest mean in all 72 cells; strictly higher in 71
Equal-allocation replay, moderate tool-free pressure	exceeds contest performance for four of six models
Fixed schedulers, strong agentic pressure	at least one exceeds the contest mean in six of nine cells
Strategy updating	limited; failure patterns are pressure-dependent; no policy dominates across domains
The sentence to keep is the second row. Splitting the budget six ways with no
reasoning at all beats the model's own reasoning about how to split it, for four of
six models. The other three mechanisms all propose machinery; this baseline is free.

It converges with the same-day negative result on Eval Harness Configuration. How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905) names the faculty missing across all 8 harness-model combinations it tested as a metacognitive loop — checking output against evidence, revising, questioning the path. Allocating a shared budget requires judging your own likely success on each problem before spending, which is that faculty priced in tokens. Neither paper cites the other; the pairing is this wiki's and is labelled as such. It also restates Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning (arXiv:2607.29211)'s systematic miscalibration from the allocation side rather than the refusal side.

Consequence for how this wiki reads a reasoning score. Every such figure here comes from independent per-task budgets. R³-Bench says that regime overstates what the same model does when it must allocate — the condition an agent actually runs under. Nothing read quantifies the offset, and no source attempts one, so no figure on any model page is adjusted; this is recorded as a known bias in the measurement, not as a correction to apply.

Unresolved and material: the abstract gives a direction and never a magnitude ("strictly higher in 71 of 72" is satisfied by a uniform 0.3 points or by 15), and the budget's unit is not stated — which makes it incomparable to the token-count claims in the mechanisms below, the same defect Eval Harness Configuration tracks.

2026-08-18 — a third mechanism, and it argues the right budget is sometimes zero

Yesterday's two mechanisms both assume the model will eventually answer: Thought-Level Beam Search for Reasoning (arXiv:2608.08020) moves compute between live traces, Full-bandwidth transformer (arXiv:2608.08888) reduces what a step needs. Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning (arXiv:2607.29211) is categorically different — it says a substantial share of inference spend goes on questions where no amount of reasoning produces an answer, and the model does not know it (source).

Its diagnosis, stated without figures: universal capability overreach, systematic miscalibration between capability and behaviour, and a dominant failure mode of specious reasoning — output that looks valid and contains subtle errors — which escalates with task difficulty. Its fix, CaRL, shapes reward to incentivise refusal over futile reasoning and converts past failures into refusal supervision.

Mechanism	What it changes	Answer still produced?
Thought-Level Beam Search for Reasoning (arXiv:2608.08020)	allocation across live traces	yes
Full-bandwidth transformer (arXiv:2608.08888)	what one decoding step needs	yes
Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning (arXiv:2607.29211)	whether to start at all	no — refusal is the output
SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning (arXiv:2608.14277)	where a student learns when to stop	yes
The fourth row is the day's quieter finding. SimpleOPD masks the advantages of
termination tokens — </think>, `<	im_end	>` — because a short-context student
distilling from a long-context teacher inherits the teacher's stopping behaviour
and blows up its own response length. **Stopping is a learned behaviour that
transfers badly**, which is the same phenomenon CaRL treats as miscalibration,
observed inside a distillation pipeline instead of at inference
(source).

The ceiling is not additive. Four mechanisms all claiming to remove wasted tokens compose only if the tokens they remove are disjoint, and nothing read establishes that any two of them are. None of the four cites another.

The caution this page records against CaRL specifically: an objective that pays a model to refuse has an obvious degenerate solution, and the abstract's only defence is "without sacrificing utility" with no figure attached. A refusal rate is the single number needed to read the result, and it is not published.

2026-08-17 — the waste got measured, from both ends, on the same day

This page's open problem has been "how much inference cost can users absorb". Two things arriving together reframe it: a large share of that cost may be spent on trajectories that were already lost, in which case the question is not absorption but waste.

The cost, measured by a practitioner. Running Qwen 3.8 27B locally as a 17GB GGUF on an M5 Max MacBook Pro, Simon Willison recorded a single "generate an SVG" prompt consuming 22,276 reasoning tokens against 3,223 output tokens — 6.9 reasoning tokens per output token — in 21 minutes. That is the behaviour of the model's shipped default: Qwen3.8 ships reasoning_effort at xhigh, with medium and low available and not chosen for the user. His own summary is that the model "will happily burn 20,000 reasoning tokens on a two-sentence answer" (source).

The recovery, claimed by an algorithm. Thought-Level Beam Search for Reasoning (arXiv:2608.08020) argues the critical question has shifted "from how much compute to spend, to where to allocate it", and reports up to 68.5% fewer total tokens than standard parallel sampling with accuracy up — +6.7 absolute on HMMT-24, +3.3 on AIME-25 — and >2× throughput, under identical hardware (source).

The two are not about the same model and neither cites the other; the pairing is this wiki's. But a token count and a claimed recovery rate have never sat on this page together before, and they point the same way: the marginal reasoning token is not the one being priced.

A third result in the same batch attacks the same waste before inference. Full-bandwidth transformer (arXiv:2608.08888) feeds the previous top-layer hidden state back into the next decoding step, and reports shorter reasoning traces at equal or better accuracy — the implication being that part of a chain of thought exists to carry computation between steps, not to perform it, and that a wider vertical channel makes those tokens unnecessary (source).

The consequence for reading a Pricing cell. Three mechanisms — reallocation between traces, a wider latent channel, and a default effort level — each change the token count for the same answer without changing the price per token. A model page's Pricing row is a rate; none of these is visible in it. That is the same observation Eval Harness Configuration makes about benchmark scores, arriving at the invoice instead of the leaderboard.

Scaling Laws (open question)

• Does test-time compute follow power-law scaling like training compute?
• Where is the saturation point?
• Does scaling differ by domain (math vs coding vs science vs real-world)?

→ Achieving Gold-Medal-Level Olympiad Reasoning via Simple and Unified Scaling provides the most recent data in this area

Open Problems

• Cost-quality trade-off — how much inference cost can users/products absorb?
• Latency — models that think more at inference respond slower. Which use cases can tolerate it?
• Verifier accuracy — the verifier's own limits become the new ceiling
• Compute allocation optimization — how to distribute the same budget across step / sample / search?

Key Papers

• Achieving Gold-Medal-Level Olympiad Reasoning via Simple and Unified Scaling — 2026-05
• BDH-CQ: In-Context Learning with Recurrent Latent Reasoning (arXiv:2608.09888) — 2026-08, test-time compute spent in recurrent latent state instead of decoded tokens: a 150M model reports 29.5% pass@2 on ARC-AGI-1 at a computed $0.0007 per task, claimed as a new point on the benchmark's cost-accuracy frontier rather than its accuracy leaderboard (source)
• Thought-Level Beam Search for Reasoning (arXiv:2608.08020) — 2026-08, allocation rather than budget: thought-level beam search pruning and re-branching mid-run, +6.7 HMMT-24 / +3.3 AIME-25 absolute over pruning baselines with >2× throughput and up to 68.5% fewer tokens (source)
• Full-bandwidth transformer (arXiv:2608.08888) — 2026-08, a wider vertical channel between decoding steps; shorter reasoning traces at equal or better accuracy, and a 1B model matching one trained on ~1.5× more tokens (source)
• Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning (arXiv:2607.29211) — 2026-07, futile reasoning: universal capability overreach and systematic miscalibration, with specious reasoning as the dominant failure mode; CaRL shapes reward toward refusal and converts failures into refusal supervision. No figures published (source)
• R³-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets (arXiv:2608.16033) — 2026-08, allocation under a shared budget: an offline oracle built from the same model's single-problem curves matches or exceeds contest performance in all 72 cells and is strictly higher in 71; equal-allocation replay beats the model's own choices for four of six models (source)
• SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning (arXiv:2608.14277) — 2026-08, tokenizer-agnostic on-policy distillation; masks the advantages of termination tokens because a short-context student inherits a long-context teacher's stopping behaviour. Intern-S2-Preview +21.2 on ProofBench, to 55.2 (source)

Related Concepts

• Reasoning Models — the class of models that use this mechanism
• scaling-laws — separate page TBD
• Agents (LLM Agents) — multi-step agents are also a form of test-time compute

Open Debates

• "Simple scaling" camp (suggested by Achieving Gold-Medal-Level Olympiad Reasoning via Simple and Unified Scaling) vs "specialized methods" camp — which side drives reasoning progress?
• Test-time vs train-time compute allocation — if pre-training scaling saturates, where does it go?

Notable Statements

• (Sam Altman): the "automated AI research intern by 2026-09" goal presupposes strong test-time compute + an agentic loop