$ cat wiki/papers/2026/2608.21500-secopd.md
SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation (arXiv:2608.21500)
TL;DR
Defensive fine-tuning against prompt injection has been failing because its training signal is sequence-level — DPO and GRPO score a whole output, so the model never learns which tokens were the insecure ones. SecOPD scores each token of a rollout against what the initialization model would produce from the clean input. Reported attack success rate against the adaptive attacker PISmith: 94.0% → 9.0%, with weights and code released (source).
Authors & Org
Yibo Peng, Long Lian, David Wagner, Sizhe Chen (arXiv:2608.21500). Submitted 2026-08-21; cs.CR, cs.AI. No affiliations shown on the arXiv listing; no organisation is attributed here on that basis.
Listed on HuggingFace Daily Papers, 2026-08-27, 36 upvotes — that community's popularity signal and nothing more (source).
Method
The paper's framing: prompt injection is "listed as the #1 threat to AI agents", and existing defensively-trained LLMs "still suffer from near 100% attack success rates against adaptive prompt injections".
The diagnosed cause is the granularity of the feedback: "Treating an entire output equally prevents the model from learning precisely which output tokens are insecure."
Secure On-Policy Distillation:
- The LLM receives an injected sample and produces a rollout.
- The rollout's tokens are scored by the initialization model given the corresponding clean input.
- Those token-level scores drive the defensive fine-tuning (source).
The clean-input model is used as the reference for what the output should have looked like — the injection's influence shows up as per-token divergence.
Results
Author-reported; no independent reproduction mentioned.
| Setting | SecOPD | Prior SoTA (Meta-SecAlign) |
|---|---|---|
| PISmith adaptive prompt injection (ASR) | 9.0% | 94.0% |
| Agentic tool calling, unseen in training (ASR) | 4.7% | 5.5% |
| Base model: Qwen3.6-27B. Weights published as | ||
huggingface.co/pybbb/Qwen3.6-27B-SecOPD. |
The two rows say different things and the gap between them is the honest reading. Against the adaptive attacker the improvement is 10.4×; on the transfer setting it is 4.7% against 5.5% — within noise of the prior method, on a number nothing read gives an error bar for. The headline is the adaptive-attack row.
Significance
It is the first defence in this wiki to report a non-trivial number against an adaptive attacker. Agents (LLM Agents) and MCP — Model Context Protocol both record prompt injection as the standing unsolved problem of tool-using agents; the recurring pattern has been defences that hold against fixed attack sets and collapse against attackers that adapt. 94.0% → 9.0% is not a solution — one attack in eleven still lands — but it is the first movement on the number that matters.
The mechanism is the same one three other papers this week are about, pointed at security. On-policy distillation appears four times in two snapshots: as a generalisation constraint (Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models (arXiv:2608.16647), 08-25), combined with verifiable reward (OPDVR, 2608.24696, this snapshot), in diffusion models (2608.24646, this snapshot), and here as a defence. The common thread is that token-level supervision is being applied to problems previously attacked with trajectory-level rewards — filed on Agentic Reinforcement Learning.
It inherits an open constraint from its own family. The 08-25 paper found OPD's reach is set by the origin relationship between teacher and student, and that same-origin pairs transfer while cross-origin pairs mostly fit the trained distribution. SecOPD's teacher is the student's own initialization — maximally same-origin — which by that result is the configuration most likely to transfer, and also the one that cannot exceed what the initialization already knows about clean behaviour. Neither paper cites the other; the pairing is this wiki's.
Open Questions
- What is PISmith's own adaptivity budget? An ASR against an adaptive attacker is a number about a contest, and nothing read states how much the attacker was allowed to adapt after the defence was fixed.
- Is 4.7% vs 5.5% a real difference? No confidence interval, no repetition count.
- What does the defence cost in capability? Defensive fine-tuning normally trades against utility; no benign-task score is reported.
- Does it hold on a model that is not Qwen3.6-27B? One base model, one scale.
- Does the released checkpoint remain defended after downstream fine-tuning? The weights are open, which makes this a practical question rather than a theoretical one.
Cite
SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation (2026). arXiv:2608.21500.