$ cat wiki/papers/2026/2608.15008-harness-the-memory.md
Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008)
TL;DR
Runs every common memory substrate for LLM agents through one unified harness — 26 metrics, 3 backbone models, 4 benchmark suites — and reports that no substrate consistently dominates. The finding with teeth is directional rather than a ranking: broad retrieval helps long-context factual QA and can harm sequential decision-making, by pulling attention away from action-critical context (source).
Authors & Org
Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the
paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date
and abstract only
(source).
Listed on HuggingFace Daily Papers, 2026-08-20, 11 upvotes — that community's popularity signal and nothing more (source).
Method
The object of study is the memory substrate: the underlying medium in which an agent's memory is represented and stored, as distinct from the retrieval policy over it.
| Component | What it is |
|---|---|
| Substrates compared | dense and sparse indices, text records, structural stores, hierarchical stores, refinement-based memories, parametric updates, activation-compatible context mechanisms |
| Backbones | 3 models |
| Benchmarks | 4 suites, spanning user-centric question answering and agent-centric decision-making |
| Instrumentation | 26 performance and efficiency metrics under a unified harness |
| Code is stated to be released upon acceptance — i.e. not yet. |
Results
- No single substrate consistently dominates across the four suites.
- Broad retrieval benefits long-context factual QA.
- Excessive retrieval can harm sequential decision-making, the stated mechanism being that it shifts attention away from action-critical context.
- Scalability is a further routing axis: substrates that perform well at moderate history lengths can become costly or brittle at longer horizons.
- The paper's proposal is therefore substrate routing — selecting the substrate per operating regime — as a necessary component of adaptive agent memory.
What the abstract does not give: which three backbones, which four suites, any score, any per-substrate ranking, the 26 metrics, or where the "moderate" / "longer" horizon boundary sits.
Significance
The regime-dependence is the point, and it is the second such result in one day. Demystifying Agent Skills: Why They Work — Until They Don't (arXiv:2608.14036) reports that skill retrieval precision collapses 29.6% → 3.3% as the pool grows from 5 to 100; this paper reports that more retrieval actively hurts sequential decision-making while helping factual QA. Both say the same structural thing from different sides: retrieval quantity is not monotone in usefulness, and the task class decides the sign. Neither cites the other, and the pairing is this wiki's.
It is also the clean methodological counterpart to Eval Harness Configuration's standing complaint. That page's objection is that agentic results are reported as one number from one unnamed configuration. This paper holds the harness fixed and varies only the substrate — the shape of experiment the page has been asking for — and the answer is that the winner changes with the regime. That is a harder result to build on than a ranking would be, and more useful: it says a memory benchmark leaderboard would be measuring its own task mix.
The unresolved tension with the harness cluster is worth stating plainly. StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089) gets its gains from durable state — remembering more, structurally. This paper finds that remembering more degrades exactly the class StateM targets: sequential decision-making. The two are reconcilable if organised, phase-local state and broad retrieval are different operations, which is what StateM claims, but nothing in either paper tests that directly.
Open Questions
- Where is the harm threshold? "Excessive retrieval" is the finding's hinge and the abstract gives no quantity, no k, and no curve.
- Is substrate routing measured or proposed? The paper motivates routing as a necessary component; nothing read says it implements or evaluates a router.
- Do the three backbones agree? A substrate ranking that flips between backbones would be a stronger claim than the paper makes, and one that holds across them would be a weaker one. Neither is stated.
- Parametric updates in the same table as dense indices — these differ in cost by orders of magnitude, and the efficiency half of the 26 metrics is where that should show. No efficiency figure was published in anything read.
- Author list, affiliation, licence — unknown; the paper was not read.
Cite
Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents
(2026). arXiv:2608.15008.