Quick Take
- The finding: A Google DeepMind attacker built from a swarm of 16 collaborating AI agents found working prompt injections against every model it tested. Even the most robust one, Gemini 4 Argon, resisted only 66% of scenarios at the full attack budget.
- Why it matters: Human red-teaming on the public GraySwan arena is close to saturation, with recent Gemini models scoring under 1% on its transfer attacks. This report argues the plateau is misleading: stronger automated attackers still get through.
- What it means: Attacks scale with attacker capability. Moving the attacker from Gemini 3.6 Flash to Gemini 4 Argon cut average defender resistance from 42.1% to 23.1%. The authors conclude defenses must not rely on model hardening alone.
Prompt injection is the attack where text hidden in a web page, email or tool result tells an AI agent to do something its user never asked for. By late 2026, frontier labs report that their newest models shrug off most known attacks. A technical report from Google DeepMind, “Scaling Adaptive Prompt Injection Attacks with Agent Swarms” by Alexey Kurakin and ten colleagues, tests that claim against a much stronger adversary: a team of AI agents that share notes while hunting for an injection that works. The answer is that prompt injection is “quite far from solved.”
What the study shows
- No defender is fully robust: Against the strongest attacker (Gemini 4 Argon), Gemini 4 Argon as defender resisted 87% of scenarios within 150 queries and 66% at the full budget. Gemini 3.8 Flash resisted 10%, Gemini 3.7 Flash 6% and Gemini 3.6 Flash 1%.
- Open-weight models are easy targets: Kimi K3, GLM-5.3 and DeepSeek-V4.1-Flash each resisted just 1% of scenarios at the full budget, despite strong scores on standard capability benchmarks.
- Better attackers, weaker defenses: With Gemini 3.6 Flash as attacker, Gemini 4 Argon resisted 100% of attacks within 150 queries. With Gemini 4 Argon attacking itself, that fell to 87%. Mean resistance across all seven defenders dropped from 42.1% to 23.1%.
- Shared memory helps the attacker: When the strategy database was updated during the run, a Gemini 4 Argon attacker pushed Gemini 3.8 Flash’s resistance down to 5%, compared with 34% with a frozen database.
- Visible reasoning leaks: When the attacker could read the defender’s chain of thought, resistance fell, for example from 100% to 92% for Gemini 4 Argon and from 73% to 61% for Gemini 3.8 Flash.
- Beats the open baseline: On 80 AgentDojo scenarios against Gemini 3.6 Flash with Vertex AI’s built-in defenses on, the open-source Prompt Siren attack broke 2. The swarm approach, with GLM-5.3 as attacker, broke 6 within 25 queries.
| Defender | Resisted within 150 queries | Resisted at full budget |
|---|---|---|
| Gemini 4 Argon | 87% | 66% |
| Gemini 3.8 Flash | 34% | 10% |
| Gemini 3.7 Flash | 31% | 6% |
| Gemini 3.6 Flash | 3% | 1% |
| Kimi K3 | 5% | 1% |
| GLM-5.3 | 1% | 1% |
| DeepSeek-V4.1-Flash | 1% | 1% |
Why it matters: an attacker that scales with model capability
Most prompt-injection benchmarks use fixed payloads, which frontier models saturated by mid-2025. Adaptive attacks exist, but the authors note that almost none can be reproduced on new models because code and attacker models are rarely shared. This report adds two things. First, a dataset whose attacker goals are harmless but unrelated tool calls, so it measures whether a model respects the instruction hierarchy rather than whether it recognises something as dangerous. Second, an attacker design whose strength grows with the model behind it. The grading is strict: an attack only counts if the model calls the exact target tool with all arguments matching, which rules out partial or lucky hits. The same pipeline is run across seven defenders and four attackers, so the trend with attacker strength is visible in one table.
Method in brief
- Dataset: 100 synthetic tool-use scenarios built from 249 tools across 53 domains, generated and validated with Gemini. Each pairs a user task in one domain (for example tutoring) with an attacker goal in another (for example logging health data). The injection is placed in the last tool result.
- Attacker (“Agentic Optimizer”): 8 teams of 2 agents each, every agent limited to 20 evaluation calls, so 320 attempts per scenario at most. Agents share a team whiteboard, read team and global leaderboards, and start from a database of strategies that worked before.
- Models: Gemini 3.6 Flash, 3.7 Flash, 3.8 Flash and 4 Argon, plus Kimi K3, GLM-5.3 and DeepSeek-V4.1-Flash via hosted endpoints. Gemini ran without safety classifiers and with out-of-model prompt-injection defenses switched off. Defenders used high thinking, hidden from the attacker unless stated.
- Metric: Attack Resistance Rate (ARR), the share of scenarios where no successful injection was found, reported at 150 queries and at the full budget.
What remains open
- Defenses switched off: Gemini was tested without its external classifiers and out-of-model protections, so the numbers describe the model alone, not Gemini as deployed in products.
- Synthetic, benign goals: The scenarios are generated by Gemini and the attacker goals are harmless tool calls. The authors argue chains of benign calls can add up to real harm, but that step is not tested here.
- No error bars, no code: The report gives no confidence intervals or repeated runs, and neither the dataset nor the attacker is released, so outside groups cannot reproduce the numbers.
- Not peer reviewed: This is a technical report on arXiv. All eleven authors work at Google DeepMind, which builds the Gemini models under test.
- The link to cyber skills is suggestive: Attacker strength roughly tracks ExploitBench cyber scores, but the authors call this an area for future work.
Context: from static payloads to attacker swarms
The report builds on a line of adaptive-attack work: “The attacker moves second” (Nasr et al., 2025) pushed success rates against twelve defenses from nearly 0% to over 90%, and RL-Hammer (Wen et al., 2025) trained attackers that hit GPT-5 and Gemini 2.5. On defense, the authors point to model hardening, input filters and system-level designs such as CaMeL and FIDES. Our earlier coverage of an Amazon study on task tampering found a related gap: defenses tuned to known attacks miss new ones. This report adds that the gap widens as attacker models get better.
What it means for teams deploying agents
- Assume the model can be talked into it: Even the most robust model tested failed on about a third of scenarios at full budget. Limit what tools an agent can call after reading untrusted content.
- Layer defenses: The authors recommend a full stack with defenses in depth: classifiers, permission checks and system designs that separate data from instructions.
- Hide reasoning from untrusted parties: Visible chain of thought made attacks easier in this study.
- Red-team with strong models: Weak automated attackers can give a false sense of safety. Re-test whenever a stronger attacker model becomes available.
More research on our Safety & Alignment studies page, in our Google coverage and on the Gemini hub.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of Kurakin et al. (2026), “Scaling Adaptive Prompt Injection Attacks with Agent Swarms,” Google DeepMind technical report, arXiv:2611.11861v1 (cs.CR, submitted November 9, 2026), including the dataset appendix. All numbers come from the paper. The chart was redrawn by i10X from Table 2; the paper is licensed CC BY 4.0. No external funding or conflict-of-interest statement is given; all authors are affiliated with Google DeepMind. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
It is notable that a lab publishes evidence that its own flagship model can still be hijacked. The more uncomfortable finding is the trend: each better model is both a harder target and a better attacker. That makes prompt-injection robustness a moving benchmark, not a box to tick at launch. Releasing the scenario set, even without the attacker, would let others check how the numbers hold for non-Gemini defenders.
