$ cat wiki/papers/2026/2608.12307-ai4ai-test-time.md
AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses (arXiv:2608.12307)
TL;DR
A stronger model builds the harness; a weaker model runs inside it — and the weaker model's score roughly doubles with no parameter updates at all. Reported average across four Theory-of-Mind benchmarks: 0.49 → 0.91 (source).
Authors & Org
Nine authors: Cheng Qian, Wenting Zhao, Liangwei Yang, Heng Wang, Jielin Qiu, Heng Ji, Silvio Savarese, Huan Wang, Shelby Heinecke. Submitted 2026-08-12 (source).
Institutions are not stated on the arXiv abstract page. The attribution to Salesforce AI Research and the University of Illinois Urbana-Champaign came from a search summary, and this page carried it as fact after the snapshot had labelled it as a summary — the hedge was dropped in transit. It is repeated here only with that label attached.
Method
Strong-to-weak scaffolding. A builder model constructs an inference-time harness for a target model. The builder uses 5% of the data as a validation set to refine that harness iteratively over multiple rounds; the finalised harness is then evaluated on the full test set (source).
The abstract attributes the gain to three mechanisms: moving unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement (source). Two of those three are formatting and dispatch rather than reasoning, which bears directly on what the number means.
The abstract names no model on either side. Which model built and which was targeted is not answerable from anything read.
Results
One number was published to search and it is an average: target-model performance across four Theory-of-Mind benchmarks rising from 0.49 to 0.91 (source).
Weaker targets gained the most (source).
Still not recorded, because the abstract does not carry them: which builder and target models, which four benchmarks, the per-benchmark breakdown, any dispersion around the average, and any baseline other than the unscaffolded target.
This page said "not read" this morning because the HuggingFace snapshot carried
only the top-ranked entry's abstract and arxiv.org is blocked from the run's
sandbox. Backfilling from HuggingFace proved impossible — by 09:30 KST the id had
dropped out of that API's 50 entries entirely, so the feed rotates within the day.
The abstract above was read from arXiv instead, and the fetch script now records
all 25 abstracts at capture time.
Significance
This wiki has spent two weeks recording harness disclosure as a defect — Eval Harness Configuration exists because GLM-5.3, Gemini 3.7 Flash and DeepSeek V4-Pro-0813 all published agentic scores with no harness attached, which makes them unreproducible. This paper measures the same variable from the other end: if a harness a stronger model wrote can move a weaker model from 0.49 to 0.91, then the unpublished harness in a vendor's benchmark row is not a methodological footnote — it is potentially most of the score.
The two readings are the same finding. A benchmark number that omits the harness omits the term this paper reports as dominant.
It also complicates the open-weights argument running through Open-Weights Policy Fight. Capability transfer without parameter updates means a released small model's ceiling is partly set by whatever frontier model is available to scaffold it — a channel no licence governs.
Open Questions
- Which models? Builder and target are unnamed in anything read, and the result means very different things if the target is a 7B or a frontier model
- Does it transfer beyond Theory-of-Mind? Four benchmarks in one task family is the entire evidence base as reported
- Is 5% validation data a fair test-time method? It is labelled data used to fit the harness, which sits somewhere between prompting and training and is not free
- What is the cost? Multiple builder rounds per harness has a price, and no compute accounting was read
- Whether the harness is human-readable — which decides whether this is a capability result or a reproducibility one