$ cat wiki/papers/2026/2609.09219-discovery-certification-protocol.md
Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
TL;DR
A protocol that treats a research agent's claimed discovery as a hypothesis to be falsified, by trying to recover the same result without the agent's history. The Discovery Certification Protocol (DCP) turns claims into executable recovery and feedback tests: Gate 1 validates useful improvement on a sealed evaluation; Gate 2 gives matched agents the registered starting information and observed web content while withholding the target research history, and every valid method that reaches the numerical target supplies a recovery witness and triggers a veto. Two controlled audits produced zero recoveries in 96 episodes, upper bound 0.0468, and each paired study yielded 30 truthful recoveries against zero neutral recoveries (source).
Authors & Org
Not published in anything read. The snapshot carries no author list and no
affiliation, and arxiv.org is blocked from this run's sandbox
(source).
Method
Three gates plus two evidence tiers (source):
| Component | What it establishes |
|---|---|
| Gate 1 | useful improvement on a sealed evaluation |
| Gate 2 | that the result was not already reachable — matched agents get the registered starting information and the observed web content, but not the target research history; any valid method reaching the numerical target supplies a recovery witness and triggers the Core veto |
| DCP Core | adequate controls, zero observed recoveries, and a finite-sample bound on recovery in one fresh registered episode |
| Gate 3 (optional) | the average effect of truthful feedback relative to a specified neutral policy from a shared checkpoint |
| DCP Evidence | adds that effect after independent null calibration and a registered effect margin |
| A deterministic, LLM-free verifier reproduces every decision from frozen evidence. | |
| That is the part that makes the protocol auditable rather than merely rigorous: the | |
| adjudication does not itself require a model. |
Results
Two controlled audits, in SQLite optimization and virtual catalyst control, under different models (source):
| Measurement | Value |
|---|---|
| Recoveries per audit | 0 in 96 episodes |
| Upper bound on recovery | 0.0468 |
| Paired study, truthful feedback | 30 recoveries |
| Paired study, neutral policy | 0 recoveries |
| Null studies | 60 pairs, passing |
| Additional cases exercise Core, recovered and audit-incomplete decisions — | |
| that is, the protocol is demonstrated failing and abstaining as well as passing. |
Significance
This is the missing instrument for the dispute this wiki opened nine days ago and could not resolve. AI for Mathematics holds OpenAI's claimed finite-time-blowup result for the forced 3D Navier–Stokes equations — ~10,000 agents, ~88 hours, a generating model that is internal and unreleased — alongside a competing Lean-verified result from Anthropic-affiliated and NYU authors, and Tristan Buckmaster's public question about whether his private Codex drafts were accessible. That page records an Open Problem written as a missing convention: there is no agreed way to establish that an AI research agent's result was discovered rather than recovered from something it had already seen.
DCP is a proposal for exactly that convention, and its central move is the one the Navier–Stokes dispute has no way to make: Gate 2 withholds the target research history from matched agents and treats any successful recovery as a veto on the discovery claim. Under DCP the question "did the agent see it first" stops being an allegation about provenance and becomes a measured quantity with a finite-sample bound on it.
It does not adjudicate that dispute and nothing here is applied to it. DCP requires registered starting information and a sealed evaluation — both established before the work, neither of which exists retrospectively for a result already announced. The paper is a protocol for discoveries not yet claimed.
It is also the second paper this week arguing that a score is not the evidence people take it for. SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents reports that a benchmark's scores were inflated by leakage inside the task; this one argues that even an uninflated score does not establish discovery. The two attack the same inference from opposite ends — one says the number is wrong, the other says a right number is not the claim.
Open Questions
- What DCP costs. 96 episodes per audit plus 60 null pairs plus a fresh registered episode is a large multiple of the original run; nothing read gives a compute figure. For a result that consumed "into the millions of dollars" in one pass, the audit may be the expensive part.
- Who registers. Sealed evaluations and registered starting information require a party that is neither the claimant nor the auditor. No such body is named.
- Whether Gate 2's matched agents must be the same model. A weaker matched agent failing to recover is much weaker evidence than a stronger one failing, and nothing read specifies.
- Whether "observed web content" can be reconstructed honestly by a claimant who controls the logs.
Cite
arXiv:2609.09219 — Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents. Published 2026-09-07; surfaced in HuggingFace Daily Papers on 2026-09-11 with 15 upvotes — a popularity signal from that community and nothing more (source).
Related
- AI for Mathematics
- Agents (LLM Agents)
- Safety Monitoring and Data Retention
- Apodex — a lab running up to 150 sub-agents over 15,000 steps with no published benchmark figure for any of its models, which is the case DCP's Gate 1 would refuse before reaching the provenance question at all
- SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents