$ cat wiki/papers/2026/2608.11341-apodex-discovery.md
Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence (arXiv:2608.11341)
TL;DR
Proposes evaluating AI on real-world problems that do not arrive in executable or verifiable form, via a "heavy-duty solver" — foundation model plus harness, tools and control policies — run against environments built from a survey of 561 industries across 16 sectors, from which 423 high-value problems were assembled and 20 released. Scoring is by HDS6, which rates Tools, Repair, Alternatives, Coherence, Evidence and Scope independently of final-task success. Reported: AAV capsid design 7% above the published state of the art, and biomedical environment gains of +2.5 and +7.6 mean normalised prediction points for GPT-5.5 and GPT-5.6-sol over the same closed-book backbone (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only. No lab or author is named in anything read
(source).
Listed on HuggingFace Daily Papers, 2026-08-18, 23 upvotes — that community's popularity signal and nothing more (source).
Method
The framing: frontier models "can solve difficult tasks once the problem, tools, and success criteria are specified", while consequential real-world problems "rarely arrive in an executable or verifiable form" (source).
Three components:
| Component | What it is |
|---|---|
| Problem scouting | surveyed 561 industries across 16 sectors; assembled 423 problems; released 20 |
| Environment abstraction | a common environment–task–episode interface supplying data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts as well as final submissions |
| HDS6 | rates Tools, Repair, Alternatives, Coherence, Evidence, Scope — six axes, scored independently of whether the task succeeded |
| The unit under test is called the heavy-duty solver: foundation model, | |
| harness, tools and control policies together, pursuing "extended, stateful, | |
| verifiable investigations". The episode interface is named TRACES and is | |
| described as fixed, which is what the paper says lets a performance | |
| difference be attributed to a specific solver component. |
Results
| Domain | As reported |
|---|---|
| AAV capsid design | +7% over the published state of the art, across viability, tropism, structure prediction and generative design |
| Drug repurposing / reformulation, GPT-5.5 | +2.5 mean normalised prediction points over the same closed-book backbone |
| Drug repurposing / reformulation, GPT-5.6-sol | +7.6 points, same comparison |
| Controlled ablations are stated to show that the fixed TRACES interface **enables | |
| attribution** of performance differences to specific solver components. |
What the abstract does not give: the 20 released problems, the HDS6 scoring scale, any HDS6 score, the harness or tools used, or what "the published state of the art" was in the capsid task.
Significance
Two things here matter to this wiki, and they pull in different directions.
First, it is the cleanest statement of the harness thesis yet. The unit evaluated is explicitly not the model — it is model plus harness plus tools plus control policies, and the ablations exist to apportion credit inside that bundle. Eval Harness Configuration has been assembling exactly this claim from seven papers that each demonstrated it obliquely; this one takes it as the premise and builds the measuring apparatus around it.
Second, it is a partial answer to the comparability complaint that page carries. HDS6 scores process axes independently of final-task success, which is the same move Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417) makes on the same day from the opposite direction — that paper measures process because final scores hide bottlenecks, this one because the problems have no clean final score to begin with. Two papers, no mutual citation, converging on process metrics as the unit; the pairing is this wiki's and is labelled as such.
The +2.5 and +7.6 figures are the argument's own weakest link and its most
useful data point at once: the same environment improves two backbones by
3× different amounts. If the harness were the capability, the gain would not
be that model-dependent — which is evidence against the strong form of the thesis
the paper is otherwise advancing.
Open Questions
- What is an HDS6 point? Six axes with no scale, no reported score and no inter-rater procedure. If HDS6 is model-judged, the reward-hacking result on How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (arXiv:2608.08975) applies directly to it.
- Why does the same environment buy GPT-5.6-sol 3× what it buys GPT-5.5? Nothing read addresses it.
- "+7% over the published state of the art" across four heterogeneous capsid measures — a single percentage over four axes needs a stated aggregation, and none was read.
- 20 of 423. How the 20 were selected from the 423 determines whether the benchmark is representative or curated toward solvable cases.
- Author list, affiliation, licence, code and data availability — unknown; the paper was not read.
Cite
Apodex Discovery: Reality Benchmarks and Environments for Evaluating and
Building Discoverative Artificial Intelligence (2026). arXiv:2608.11341.
Related
- Eval Harness Configuration
- Agents (LLM Agents)
- Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development (arXiv:2608.13417)
- How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (arXiv:2608.08975)
- GPT-5.6 Sol (and Terra, Luna)