$ cat wiki/papers/2026/2608.16391-ventor-qtest.md
Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (arXiv:2608.16391)
TL;DR
A black-box audit of what a hosted API actually serves you, requiring no logprobs from the target. It reports two statistics — average fidelity loss (AFL) and extreme fidelity loss (EFL) — and finds that neither has much detectable association with GPQA-Diamond accuracy, while pronounced EFL coincides with a declining Terminal-Bench pass rate as task exposure increases (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only. The stated code location is
github.com/Tencent/AI-Infra-Guard, which was not read — github.com is under the
same block — and which is the only affiliation signal in anything read
(source).
Listed on HuggingFace Daily Papers, 2026-08-19, 12 upvotes — that community's popularity signal and nothing more (source).
Method
The stated problem: third-party providers deploying open-weight models are now an important part of the ecosystem, and auditing the quality of their inference APIs is an open problem. Hosted model routing is formalised as a stochastic process, and the audit is composite and black-box — it requires no probability information from the target.
| Component | What it measures | How |
|---|---|---|
| Repeated-request | AFL — average fidelity loss | sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from returned text counts, reports a null-bias-corrected, within-window mean coarsened-KL statistic |
| Long-sequence | EFL — extreme fidelity loss | independent runs; the empirical upper tail of a run-level reference-centered-surprisal statistic |
Results
All figures as reported (source).
| Finding | As reported |
|---|---|
| AFL validity | across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator |
| EFL variation | across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation |
| Association with accuracy | AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy |
| Association with long-horizon tasks | pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases |
| The paper's own proposed explanation for the split: correctness in long-horizon | |
| tasks may be more sensitive to extreme fidelity loss. Its recommendation is to | |
| report AFL and EFL jointly, particularly when auditing long-horizon agentic | |
| tasks. |
What the abstract does not give: the routes or providers audited, the models behind them, any AFL or EFL value, the size of the Terminal-Bench decline, or a licence.
Significance
This is the instrument Model Routing has been missing, and it
arrives two days after that page's largest open question. On 2026-08-17 Stripe
acquired OpenRouter for more than $7 billion, and this wiki recorded that
nothing read stated any intended change to the catalogue this repo's own
scripts/spec-check.py reads daily. The deeper exposure was never the catalogue's
prices — it is that a routed request's provider is not the vendor, which is the
finding that forced spec-check to compare against first-party endpoints rather than
headline prices in the first place. Ventor-QTest measures the consequence directly:
what you are actually served, per route, without the provider's cooperation.
The GPQA result is the one that should change behaviour. A route can show pronounced fidelity loss and score normally on a single-turn knowledge benchmark. Every open-weight figure this wiki records from a third-party host is a single-turn-shaped number, which means the standard evidence is structurally blind to the defect this paper measures. That bears directly on Qwen 3.8 27B, whose Artificial Analysis Intelligence Index of 52 — recorded today — is a leaderboard figure produced through some serving path nobody has named.
And it lands squarely on the harness cluster. The failure shows up on Terminal-Bench, worsening as task exposure increases — the same benchmark StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089) reports 95.3% on, and the same long-horizon regime. Two same-day papers therefore give two different reasons a long-horizon agent number moves without the weights changing: the harness around it, and the route serving it. Eval Harness Configuration's standing complaint is that a reported score does not name its configuration; this adds a component to the configuration that no paper in that cluster names at all.
Open Questions
- Which routes and providers? Seven route snapshots, three logprob-capable conditions, no names. The result is unactionable for any specific host, which for an auditing tool is the point of it.
- How large is the Terminal-Bench decline? Stated as coinciding with pronounced EFL, with no figure — a direction, not a magnitude.
- What causes fidelity loss? Quantisation, batching, speculative decoding, silent model substitution and sampling-parameter drift all fit; the method detects the divergence without identifying it, and nothing read distinguishes an honest engineering trade-off from a substitution.
- Is "little detectable association" a null result or an underpowered one? Seven routes is a small sample for a route-level association.
- Author list, affiliation, licence — unknown; the paper was not read, and the Tencent repository path is the only signal.
Cite
Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs (2026).
arXiv:2608.16391.