AI Security Agents vs. SAST: Snyk Benchmark Finds They Catch Different Bugs

i10X Quick Take

Quick Take

1. The test: Snyk researchers ran their own static analyzer, Snyk Code, against Claude Code, OpenAI’s Codex Security and the open-source DeepSec on 20 web apps with 108 exploit-verified vulnerabilities.
2. The result: Snyk Code found 82.4% and Claude Code with Opus 5 at maximum effort 79.3%, a statistically indistinguishable gap. Codex Security pinpointed at most 41.2% at the exact code location, although it often found the right file.
3. The real lesson: The two leaders catch different bugs. Together they cover 101 of 108 vulnerabilities, while the best AI-only pair reaches 89.

Can an AI coding agent replace a classic static application security testing (SAST) tool? A new preprint from Snyk, “How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0” by Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth and Brian Clark, argues that the answer is neither yes nor no: the tools split the work by bug type. One caveat comes first. All five authors work at Snyk, the vendor of one of the tested products, and the benchmark was built by that team.

What the study shows

The authors compared nine configurations in 180 runs in early September 2026: Snyk Code; Claude Code 2.1.251 with Claude Opus 5 or Sonnet 5 running its /security-review command; OpenAI’s Codex Security 0.1.24 with three GPT-5.6 variants (Luna, Terra and Sol); and DeepSec 2.3.7. A finding only counted if it named both the source and the sink of the vulnerability within two lines.

• A statistical tie at the top: Snyk Code matched 89 vulnerabilities, Claude Code with Opus 5 at “xhigh” effort 85. They disagree on 28 (16 only Snyk, 12 only Opus; McNemar p=0.57). In resampled corpora, Snyk Code ranked first 74% of the time.
• Precision differs sharply: Opus 5 xhigh had 74.6% precision (85 of 114 findings matched), Snyk Code 20.9% (89 of 425). But 87% of Snyk’s unmatched findings were outside the 14 benchmarked classes, mostly 195 hardcoded-credential alerts. Counting only in-scope findings, precision was 69% for Snyk and 81% for Opus.
• Different strengths: Snyk Code matched all 11 open redirects, all 6 prototype-pollution bugs and all 16 path traversals, but 0 of 3 authorization flaws (IDOR, mass assignment). Of the 12 bugs only Opus found, 7 were cross-file; of Snyk’s 16 extra finds, only 3 were. On same-file flows Snyk led 40 to 32 of 46; on cross-file flows Opus led 53 to 49 of 62.
• Harness policy matters: Claude Code’s /security-review prompt excludes ReDoS and restricts prototype pollution and open redirect. On those 20 bugs Snyk matched 18 and Opus 9. In a supplementary run on 19 apps without that built-in policy, Opus found 16 of these 20 (p=0.016) and 93 of 105 vulnerabilities overall, more than Snyk Code’s 87, at 0.63 times the cost.
• Codex Security finds files, not lines: With GPT-5.6 Sol, it located 80 vulnerabilities at the file level but only 34 at the line level, typically pointing a median of six lines above the sink, such as the function header.
• New bugs: The AI tools reported seven real vulnerabilities outside the reference set, all reproduced live. Codex Security reported all seven, Claude Code six and DeepSec three.

Why it matters: Coverage, not a winner

Security teams are under pressure to replace rule-based scanners with AI agents. This benchmark suggests that dropping either side leaves gaps. Snyk Code plus Opus 5 xhigh covered 101 of 108 vulnerabilities, as many as all seven comparable configurations combined, while the best pair without SAST (Opus 5 xhigh plus Codex Security with Terra) stopped at 89. The paper also shows that a harness prompt can quietly decide which bug classes an agent ignores, so “the model is weak at X” is sometimes really “the prompt told it to skip X”.

Cost and speed separate the tools even more. Snyk Code scanned all 20 apps in 8 minutes; AI configurations needed 2.4 to 8.9 hours, and DeepSec with Opus 17.8 hours. Per matched vulnerability, that is 5 seconds versus 107 to 942 seconds.

Method in brief

• Corpus: 20 open-source web apps (Go 6, JavaScript/TypeScript 7, Python 5, Java 2) with 108 vulnerabilities in 14 classes, 62 of them cross-file and 447 flow locations. Each vulnerability has an exploit proof of concept verified by the authors; many were seeded into the apps with AI coding tools.
• Scoring: Endpoint rule with source and sink within plus or minus two lines; macro recall averaged over apps; 20 alternative scoring rules as robustness check.
• Runs: One run per cell; Snyk Code was byte-identical across repeats. Total known spend was $1,594.08. Code and all run data are public on GitHub.

What remains open

• Conflict of interest: The authors are Snyk employees, and Snyk Code is the only SAST engine tested. Five apps had appeared in Snyk demos; without them, the result is 74 for Snyk versus 72 for Opus out of 87.
• Seeding bias: Seeded bugs may favor either side, as the authors acknowledge.
• Single runs: AI agents are non-deterministic, yet each configuration ran once per app.
• Codex settings: Codex Security ran in standard, not deep, mode, and covered some apps only partially (7, 6 and 3 apps for the three variants).
• Narrow scope: Java coverage is thin, and several apps are intentionally vulnerable demo projects.

Context: Coding agents move into security review

Anthropic ships /security-review as part of Claude Code, and OpenAI offers Codex Security as a dedicated product. Vendor benchmarks like this one are useful because they publish run data, but they should be read next to independent evaluations. For more on how AI systems are tested, see our Benchmarks & Evals coverage and our Claude hub.

What it means for security and platform teams

• Run both: A deterministic scanner catches pattern-like bugs (redirects, traversal, prototype pollution) fast and repeatably; an agent adds cross-file and authorization flaws.
• Check the harness prompt: Read what a review command excludes before trusting its silence on a bug class.
• Budget by match, not by run: Cost per matched vulnerability ranged from $0.47 (Codex Security with Luna) and $0.54 (Sonnet 5) to $1.94 (Opus 5 xhigh), $5.94 (Terra) and $21.92 (Sol). Sol cost 35 times as much as Luna for 11 fewer matches.
• Effort beats add-ons: Opus 5 at medium effort matched 81 vulnerabilities without the Snyk MCP server and 80 with it, so the effort setting, not MCP, explained the gap to xhigh.

About the analysis

Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.

This article is based on the full text of the preprint (arXiv:2610.38617v1, submitted October 28, 2026), including its appendix. All numbers come from the paper; i10X did not rerun the benchmark. The chart was drawn by i10X from Table 4; the paper is licensed CC BY 4.0. All authors are employed by Snyk, whose product is part of the comparison. AI tools assisted with research and drafting; the selection, fact-checking and final editing were done by the i10X editorial team.

i10x Perspective

The headline from a vendor-authored benchmark would normally be “our tool wins”. This one is more interesting: Snyk’s own data shows Claude Code within statistical noise of Snyk Code and finding bugs the scanner cannot see. The practical takeaway is less about choosing a winner than about layering a fast, deterministic baseline with an agent that reasons across files, and about reading the fine print of every security prompt.