Model Routing

Definition

Choosing which model answers which request — or which step of a request — at runtime, rather than binding an application to one model. The decision is made by a component sitting between the caller and the models: a classifier, a heuristic, or a small trained router.

Two distinct motivations produce the same machinery, and this page keeps them apart because they fail differently:

• Cost routing — send the cheap step to the cheap model, escalate only what needs it. Failure mode: a task is under-served and quietly gets a worse answer.
• Safety routing — send a request that trips a classifier to a more restricted model. Failure mode: a legitimate request is downgraded. Anthropic calls this a fallback (source).

Why It Matters

Routing is where the published price of a model stops describing what a workflow costs. Once a router is in the path, the per-token rate on a model page is an input to a cost function, not the cost — and the wiki's own spec tables cannot express that.

It also relocates a safety decision. When a classifier picks the model, the classifier, not the model, is the thing that determines what a user is allowed to get — and it is usually the component with no model card, no benchmark and no version history.

State of the Art (as of 2026-08-12)

The best-known router is acquired by a payments company for $7B+ (2026-08-17)

Stripe acquired OpenRouter for more than $7 billion, against a $1.3 billion valuation set by a $113 million Series B three months earlier. Reporting puts earlier talks near $10 billion and attributes the ~30% fall to summer-2026 model price declines — a single outlet's attribution, recorded as reported (source).

OpenRouter is the most widely used instance of the pattern this page describes: a single endpoint in front of many providers, choosing per request by capability, price and availability. What the price says about the pattern is the part worth recording — a 5× step-up in three months values the routing layer, not any model behind it, and the buyer is a payments company. The two structural readings both fit and nothing read separates them: routing is being valued as metering infrastructure (who called which model, at what cost, billed to whom), or as distribution (the gateway chooses, so the gateway has the leverage). Neither is stated by either party.

This wiki is not a neutral observer of this transaction. scripts/spec-check.py reads OpenRouter's public catalogue as the daily cross-check on every Pricing and Context window cell in this wiki's model pages, and CLAUDE.md names the catalogue as the source rather than a cross-check for models whose vendor publishes no list price. An ownership change at the catalogue is a change to this repository's verification chain. Nothing read states any intended change to the catalogue, its public API or its pricing data — recorded so that if one comes, the date it was foreseeable is on the record (source).

NVIDIA NeMo Switchyard — routing shipped as open-source infrastructure (2026-08-11)

NVIDIA released NeMo Switchyard, an open-source model routing library for AI agents, on the same day as Nemotron 3.5 Lightning. It routes each prompt to a model per step of an agent workflow (source).

It ships two families (source):

• Tuning-free routers, which decide without training on workload-specific data: an LLM classifier, a stage router, and an escalation router
• Tunable routers, trained on the workload

NVIDIA's internal benchmark claim: frontier-level accuracy at nearly one-third the task-completion cost of Opus 4.8 alone (source) (VentureBeat).

Read the pairing, not just the library. NVIDIA shipped a cheap model and, the same day, the software that decides when to use a cheap model — with the comparison baseline being a competitor's flagship. The router is the argument for the model.

Anthropic's biology fallback — routing as a safety control (2026-08-07)

Claude Fable 5 routes a request to a less capable model — Claude Opus 5 — when a classifier judges it to touch safeguarded biology. Anthropic retrained that classifier's constitution and reported ~85% fewer biology-related fallbacks in testing, with the reduction in total fallback volume spread very unevenly across surfaces: ~67% Claude.ai, ~55% Cowork, ~17% Claude Code, ~7% Claude Platform (source).

That spread is the most informative routing measurement this wiki holds. It says the router was firing on consumer questions far more than on developer traffic — i.e. the misrouting was concentrated in one population, invisible in the aggregate.

Latency-and-capacity routing in agent platforms (2026-06)

Azure Agent Mesh, announced at Microsoft Build 2026, performs automatic routing based on latency and GPU availability across on-prem Windows, Windows 365 and Azure Arc edge, with a GA target of Q4 2026 (source). This is the third motivation — routing for placement rather than for cost or safety — and it predates the other two on this page.

2026-08-19 — an audit of what a hosted route actually serves you

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (arXiv:2608.16391) formalises hosted model routing as a stochastic process and audits it black-box — no probability information required from the target (source):

Statistic	How it is obtained
AFL — average fidelity loss	a frozen constrained context sent repeatedly; an output distribution reconstructed from returned text counts; a null-bias-corrected within-window mean coarsened-KL
EFL — extreme fidelity loss	independent runs; the empirical upper tail of a run-level reference-centered-surprisal statistic
Reported: AFL shows strong linear descriptive agreement with a logprob-derived
comparator across three logprob-capable route conditions; 20-run probes across
seven route snapshots reveal route-specific EFL variation; and the split
that matters — **neither statistic has much detectable route-level association with
GPQA-Diamond accuracy**, while **pronounced EFL coincides with a declining
Terminal-Bench pass rate as task exposure increases**.

This is the first instrument on this page that measures the routing layer rather than the routing decision. Everything above is about choosing a model; this is about whether the model you chose is the one answering. It arrives two days after Stripe acquired OpenRouter for more than $7 billion, and it names the exposure that acquisition made visible: a routed request's provider is not the vendor, which is the same finding that forced this repo's own scripts/spec-check.py to compare against first-party endpoints rather than headline catalogue prices.

The GPQA result should change how a third-party figure is read here. A route can show pronounced fidelity loss and score normally on a single-turn knowledge benchmark — and single-turn-shaped numbers are what most third-party figures in this wiki are. It bears directly on Qwen 3.8 27B's Artificial Analysis Intelligence Index of 52, recorded today: a leaderboard figure is produced through some serving path, and nothing states which.

What it does not give: no provider or route is named, no AFL or EFL value is published, and the Terminal-Bench decline has no magnitude. The method also detects divergence without identifying its cause — quantisation, batching, speculative decoding, sampling drift and outright substitution all fit, and nothing read separates an engineering trade-off from a swap.

Open Problems

• Nobody publishes the router's error rate. Partly addressed, 2026-08-19 — Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (arXiv:2608.16391) measures a different error than the one named here: not "was the wrong model chosen" but "did the chosen model's output distribution shift". Both are unmeasured elsewhere; neither substitutes for the other. Every claim on this page is about the outcome of routing (cost saved, fallbacks reduced). None states how often the router sent a request to the wrong model. Anthropic's per-surface spread is the closest thing to that number and it is an inference, not a measurement. Unchanged by the field's first benchmark: xRouteBench scores quality and cost, which are also outcomes (source).
• No benchmark covers safety routing. The five xRouteBench task families (generic, memory-augmented, vision, time-series, personalized) are all capability-and-cost. The fallback case Anthropic ships optimises something else entirely, and appears in none of them — so the half of this page with a live safety consequence still has no measurement at all.
• A routed benchmark result is not a model result. If frontier-level accuracy is achieved by a router over several models, the number characterises the system. This is the same reporting problem Eval Harness Configuration records for scaffolds, arriving now for model selection.
• Cost claims are baseline-shaped. "One-third of Opus 4.8 alone" compares against running the most expensive available model for every step — a baseline nobody chooses. What it saves against a competently hand-tuned assignment is unstated.
• Safety routing has no appeal path. A user downgraded by a classifier is not told which component made the decision, and the classifier has no published version.
• Router-level attacks are unexamined here. If routing decisions are driven by a classifier reading the prompt, the prompt can be written to steer it. No source read addresses this.

Key Papers

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers (arXiv:2608.06867) (2026-08-16) — the first research this page has read, and it goes after the thing the Open Problems above say is missing: comparability. It recasts routing as a sequential decision process with five named components (context encoders, model encoders, scoring functions, decision rules, learning signals), and ships xRouteBench, which scores routers jointly on response quality and inference cost, plus an open-source library of 16+ routers. Learned routers beat the strongest fixed-model baseline by 14.6% relatively (source).

Two things it does and does not settle. Evaluating quality and cost together is what makes an under-serving router legible at all — on cost alone, sending everything to the cheapest model wins. But no error rate appears in it, so the first Open Problem below survives its own first paper: xRouteBench measures outcomes, not misroutes. And nothing read names which 16 routers are included, so it cannot yet be pointed at NeMo Switchyard.

Stealing Reasoning Traces from Proprietary LLM APIs (arXiv:2608.09867) is adjacent but not about routing: it turns on reasoning traces being interchangeable across models within one provider, which is the same substitutability that makes routing possible.

Related Concepts

• Agents (LLM Agents)
• Test-Time Compute (Inference-Time Compute Scaling)
• Eval Harness Configuration
• AI Alignment
• Nemotron 3.5 Lightning
• Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (arXiv:2608.16391)