Quick Take
- The finding: Amazon Responsible AI’s new benchmark shows that indirect prompt injection is not only “ignore previous instructions.” Across 90 task-tampering cases, attacks that rewrite decision rules, available actions or world state beat classic objective overrides on undefended agents (mean ASR 21.2%, 19.0% and 18.6% vs 15.0%).
- Defense blind spot: A strong detection-and-removal defense (DRIFT-isolation) cut objective attacks to 2.4% success but left observation- and state-targeting attacks at 9.5% and 12.1%.
- Tool policies barely help: Progent-style tool allowlists still allowed 15.4% attack success, near the 17.0% undefended mean, because tampering usually keeps the tools the benign task already needs.
LLM agents in enterprise workflows read third-party email, tickets, PRs and dispute evidence, then call tools with side effects. Most indirect prompt-injection (IPI) benchmarks plant attacker instructions that redirect the agent to a new task. A preprint from Amazon Responsible AI, “Beyond Instruction Override: Benchmarking Indirect Prompt Injection with Task-Tampering Attacks,” argues that real attackers can instead corrupt the outcome of the task already underway—for example, inflating a refund while the agent still “resolves the dispute.”
What the study shows
The authors redefine IPI by authority, not wording: planted content is an attack when it tries to decide something its source is not authorized to decide. They build 90 validated cases across Productivity, DevOps and Marketplace environments (28/31/31 cases; 1,343 single-primitive attacks plus 90 compound attacks) and score whether a predefined harmful deviation occurs.
- Six influence primitives: Each case holds the harmful outcome fixed and varies which belief the planted content targets—objective b(U), decision rules b(π), consequences b(T), available actions b(A), observation interpretation b(O), or world state b(s).
- Undefended mean ASR is 17.0%: Across DeepSeek V3.1, Kimi K2.5, Claude Sonnet 4, Claude Sonnet 4.5 and GLM-5 (Amazon Bedrock, temperature 0), model ASRs range from 11.1% (Sonnet 4.5) to 23.2% (DeepSeek). Benign task success without injection averages 95.3%.
- Non-objective attacks win more often: Without defenses, mean ASR is 21.2% for rules, 19.0% for action space and 18.6% for state, versus 15.0% for objective overrides (Table 3).
- Detection defenses overfit to instruction-like attacks: DRIFT-isolation reaches 6.6% mean ASR with high utility (98.0% / 93.3% benign / attacked TSR) but drops b(U) to 2.4% while leaving b(O) at 9.5% and b(s) at 12.1%. The isolator fully strips 52.7–59.5% of carriers overall, yet 78.2–85.1% of b(U) attacks versus only 26.2–34.3% of b(O)/b(s).
- Tool-gate defenses miss value tampering: Progent stays near undefended ASR (15.4% vs 17.0%) while cutting benign TSR to 74.9%. Emitted attacked calls are blocked only 0–9.7% of the time. CaMeL reaches 1.3% ASR but collapses utility to 13.3% benign TSR.
| Defense | Mean ASR % | Benign TSR % | Note |
|---|---|---|---|
| None | 17.0 | 95.3 | b(π)/b(A)/b(s) beat b(U) |
| DRIFT-isolation | 6.6 | 98.0 | b(U) 2.4; b(O) 9.5; b(s) 12.1 |
| Sandwich | 10.4 | 98.5 | Prompt defense |
| Progent | 15.4 | 74.9 | Near undefended ASR |
| CaMeL | 1.3 | 13.3 | Strong security, poor utility |
Human case validation reached 91.9% overall agreement on the three validity conditions. Trajectory labels (Claude Opus 4.6) show Sonnet 4.5 resists planted content more often than GLM-5 (44.3% vs 25.1%) and flags manipulation especially on objective attacks.
Why it matters: IPI evals that only redirect the task flatter defenses
The benchmark keeps the benign task, attacker role and harmful outcome fixed while swapping the influence channel. That isolates whether a defense actually covers authority abuse, or merely catches instruction-shaped text. The gap under DRIFT after detection-optimized rewrites shows the blind spot is structural, not a wording quirk.
Method in brief
- Environments: Company productivity assistant, open-source maintainer bot, marketplace dispute agent—each with explicit trust hierarchies (internal records outrank party claims, and so on).
- Models: DeepSeek V3.1, Kimi K2.5, Claude Sonnet 4, Claude Sonnet 4.5, GLM-5 as ReAct agents on MCP tool servers.
- Defenses: Spotlighting, Sandwich, DRIFT-iso, DRIFT-full, Progent, CaMeL, ARGUS.
- Metrics: ASR (harmful deviation under attack) and TSR (task success with/without injection).
- Artifacts: Code and data at amazon-science/task-tampering-prompt-injection (CC BY-NC 4.0).
What remains open
- Simulated only: Fictional orgs and tools; attacks do not transfer to production without redesign.
- Utility tradeoffs: The strongest provenance defense (CaMeL) is currently too restrictive for many workflows.
- Model set: No GPT-family agent in the main five; Claude results may not generalize to every stack.
- Dual use: The paper catalogues attack forms; authors argue passive, environment-specific cases mainly help defenders enumerate surface area.
- AI-assisted construction: Claude Code helped draft cases and code; humans reviewed validity and all reported numbers are scripted from run records.
Context: From AgentDojo redirects to authority abuse
OWASP ranks prompt injection a top LLM risk. Benchmarks such as AgentDojo largely stress instruction-bearing redirects. This Amazon work sits beside that line by insisting on task tampering: the agent still pursues the user’s job while an unauthorized source rewrites amounts, policies or state. That is closer to how fraud and insider content look in customer support and DevOps. Related i10X coverage includes pretraining poisoning of agents and Claude Mythos cyber-defense field results.
What it means for agent builders and security teams
- Screen for authority, not just jailbreak phrasing: Ask whether a source may set the contested field (refund amount, merge decision, recipient).
- Do not equate tool allowlists with IPI defense: If the malicious call uses an expected tool with a poisoned argument, allowlists will mostly wave it through.
- Test observation and state attacks: If your red team only plants “ignore previous instructions,” you will overestimate DRIFT-like isolators.
- Watch utility: Ultra-strict provenance (CaMeL-style) can crush benign completion; pick defenses that keep TSR near 90%+.
- Teams shipping on Claude or other Bedrock models should add task-tampering cases to release gates alongside classic IPI suites. More on our Security & Defense hub and Anthropic coverage.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of Zhang, Bertran, Avula, Lee and Murdock (2026), arXiv:2611.07640v1 (cs.CR, submitted November 5, 2026), including Table 3 and the reproducibility statement. All numbers come from the paper; i10X did not rerun the benchmark. The preprint is CC BY 4.0; accompanying code/data are CC BY-NC 4.0. The chart was redrawn by i10X from Table 3. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team. See also Safety & Alignment studies.
i10x Perspective
Prompt injection is graduating from meme jailbreaks to quiet corruption of business decisions. If your agent can move money, merge code or change access, defenses that only hunt for “ignore the user” are solving yesterday’s demo. Authority-aware checks—and evals that reward them—are the next bar.
