$ cat wiki/papers/2026/2608.14036-demystifying-agent-skills.md

Demystifying Agent Skills: Why They Work — Until They Don't (arXiv:2608.14036)

paperupdated 2026-08-20created 2026-08-20

agents harness skills retrieval failure-analysis evals

TL;DR

Isolates what a skill actually does for an LLM agent and finds it is not what the name suggests: procedural anchoring accounts for 65.7% of skill cases against 4.5% for explicit knowledge injection — skills stabilise action, they do not supply missing facts. And they carry a separate, sharper failure: as the skill pool grows from 5 to 100, actual-use precision falls from 29.6% to 3.3% (source).

Authors & Org

Not obtainable. arxiv.org is EGRESS_BLOCKED from this environment and the paper was not read; the HuggingFace Daily Papers snapshot carries title, id, date and abstract only (source).

Listed on HuggingFace Daily Papers, 2026-08-20, 122 upvotes — that community's popularity signal and nothing more. Published 2026-08-14, so it sat six days in the daily feed before this wiki picked it up (source).

Method

The subject is skills: structured packages of knowledge attached to an LLM agent at inference time. The stated gap is that existing evaluation measures whether skills raise aggregate task success, and therefore cannot say when they help, why they work, or where they fail.

Component	What it is
Controlled experiments	across multiple benchmarks, agent harnesses and LLMs, isolating four variables: representation, outcome annotation, retrieval difficulty, cross-framework robustness
Contrastive study	controlled quantitative experiments paired with trajectory analysis
Corpus	8,135 normalised trial records; 238 valid unique labels retained from 240 open-coded records
Taxonomy	3 high-level categories and 12 skill-use modes

Results

Finding	Figure
Procedural anchoring as share of skill cases	65.7%
Explicit knowledge injection as share	4.5%
Skills over Workflow Memory, matched comparisons	+6.06 points
Actual-use precision, pool of 5	29.6%
Actual-use precision, pool of 100	3.3%
The paper's mechanism claim: skills work when **noisy trajectories become
procedural anchors that stabilise execution**. Its failure claim: skills fail
under brittle assumptions, incompatible contexts, or insufficient adaptation.

Two results about retrieval are stated together and pull in different directions, and both are recorded here rather than the convenient one alone:

Retrieval is named as a separate bottleneck — the 29.6% → 3.3% collapse.
Yet confusable distractors impair offline identification while downstream success remains stable, and exact ground-truth invocation is neither sufficient nor necessary.

What the abstract does not give: which benchmarks, harnesses or LLMs; the 12 modes; the base rates behind the +6.06; whether the precision collapse was measured on one harness or several; and what "actual-use precision" is precision of.

Significance

This is the mechanism underneath the joint reading Eval Harness Configuration adopted on 2026-08-19 — that harness scaling buys execution reliability and does not buy self-assessment. That position was assembled from two papers that measured outcomes (StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089) and How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks (arXiv:2608.14905)); this one measures the channel, and its answer matches. 65.7% procedural anchoring against 4.5% knowledge injection is a direct statement that the artefact stabilises action — which is exactly "execution reliability" and exactly not "knowing whether the output is right". Three papers, three methods, no cross-citation; the convergence is this wiki's reading and is labelled as such.

The retrieval collapse is the load-bearing new fact. Every skill and memory-substrate result in this wiki is reported at a fixed, small pool size. A precision figure that falls 9× between a pool of 5 and a pool of 100 means the published gains are gains at the pool size tested, and the deployment regime that motivates skills — an accumulating library — is the regime where the measured mechanism degrades fastest. It arrives the same day as Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008), which reports the neighbouring finding from the memory side: excessive retrieval can harm sequential decision-making. Neither cites the other.

It also weakens the "skills are portable knowledge" framing that Agents (LLM Agents) has carried since the skill format became common. If 4.5% of cases are knowledge injection, a skill library is closer to a runbook than to a knowledge base — and StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling (arXiv:2608.15089)'s runbook transferring unchanged across a model version bump is the same object seen from the other end.

Open Questions

Precision of what? The 29.6% → 3.3% figure is the paper's sharpest number and the abstract never defines the denominator. Whether it is retrieval precision against a ground-truth skill or the fraction of retrieved skills that get used changes what it implies for a deployed library.
Does the collapse bite? The paper says downstream success remains stable under confusable distractors, and that exact invocation is neither sufficient nor necessary. Those two claims and the precision collapse are hard to hold at once, and nothing read reconciles them.
Is 65.7% / 4.5% task-dependent? Both numbers come from the paper's own open-coding of its own trajectories. A benchmark suite weighted toward procedural tasks would produce this split by construction.
How many harnesses? "Various agent harnesses" is the whole basis for the cross-framework robustness claim, and none is named.
Author list, affiliation, licence — unknown; the paper was not read.

Cite

Demystifying Agent Skills: Why They Work-Until They Don't (2026).
arXiv:2608.14036.

Referenced by

Agents (LLM Agents)ASI-Bench: At the Dawn of Artificial Superintelligence (arXiv:2608.17271)Eval Harness Configuration Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents (arXiv:2608.15008)