Stripe Evals: Coding Agents Score 78% on Payment Integrations, but Only 29% Pass Every Required Check

Stripe published Stripe evals on 19 November 2026, a benchmark that tests 14 model and harness pairs, from Claude Code to Codex and OpenCode, on 69 realistic payment integration scenarios. Agents averaged 78.1%, but only 29.4% of runs met every required criterion, and Stripe's agent plugin lifted all of them.

Christopher Ort
By i10X News Desk · Reviewed by Christopher Ort · 3 min read
Share
A shop counter with a card payment terminal next to a laptop and a person's hands typing.
Image: AI-generated, i10X
On this page
  1. Quick Take
  2. In plain English
  3. What happened
  4. Background
  5. How it works
  6. What the results show
  7. Why it matters
  8. What's still open
  9. Sources
  10. i10x Perspective

Stripe has published Stripe evals, a public benchmark that measures how well AI coding agents build real payment integrations. Across 14 combinations of models and coding tools, the agents scored 78.1% on average. But when Stripe counted only runs that met every required criterion, the rate fell to 29.4%.

i10X Quick Take

Quick Take

  1. What’s new: Stripe evals and an open eval framework, published on Stripe’s developer blog on 19 November 2026.
  2. The test: 69 scenarios, 24 one-time payments, 26 subscriptions and 19 Connect marketplace cases, each run twice with and without Stripe’s agent plugin.
  3. Headline gap: the average score across all agents falls from 78.1% to 29.4% under the strict pass rate.
  4. Top strict scores: GPT-6 Astra in Codex reached 52.9% with the plugin, GPT-6.1 Sol 50.7%, and Fable 5.2 in Claude Code 43.2%.
  5. Plugin effect: all 14 agents scored higher with the plugin, by about 3 to 8 points, and web searches fell 96%.

In plain English

Stripe handles online payments for many companies, and more developers now ask AI assistants to write the code that connects their apps to it. Stripe tested how good these assistants really are. Most of their code looks mostly right, but only about three in ten attempts got every essential detail correct, and with payments a small mistake can mean money goes to the wrong place.

What happened

Stripe researchers Kiran Prasad, Yiing Chau Mak and Manish Kurse explained why they built the benchmark: agents “bias towards older APIs that are more prevalent in their pretraining corpus, miss key features, and hallucinate.” Existing coding benchmarks, they wrote, “check whether the code passes deterministic checks, not whether the API integration is correct.”

Stripe tested Claude Code with Haiku 5.5, Sonnet 5.5, Opus 5.5, Fable 5.1 and Fable 5.2, Codex with GPT-6 Luna, GPT-6.1 Sol and GPT-6 Astra, and OpenCode with Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Kimi K3, GLM 5.3 and MiniMax M3.1 Flash. That is 276 runs per agent.

Background

Vendors are increasingly writing their own tests for coding agents, because general benchmarks miss product-specific mistakes. Supabase did this for databases with Supabase Evals, and recent tests show that the harness around a model can matter as much as the model.

How it works

  • A simulated developer: a roleplaying agent starts with a short request, keeps a private list of two to four needs, and asks follow-up questions one at a time, like a real client.
  • Real workspace: coding agents could inspect the repository, write files, run commands and revise their work over several turns.
  • Grading: judges score each rubric criterion separately using the conversation and a snapshot of the code. Stripe used GPT-6.1 Sol as roleplayer and judge and ran a check with Opus as judge, which “showed no bias.”
  • Two scores: the integration success rate averages all required and recommended criteria; the strict pass rate needs every required criterion to pass.

What the results show

  • Best averages: with the plugin, GPT-6 Astra scored 90.0% and GPT-6.1 Sol 89.6%; Opus 5.5 reached 86.2%.
  • Cost differences: a run cost about $0.07 with GPT-6 Luna and $3.63 with GPT-6 Astra when using the plugin.
  • Fewer outdated calls: the plugin cut outdated parameter references from 7% to 1.3% of parameters in API calls.
  • Category swings: agents that led on payments and subscriptions did not lead on Connect, so Stripe warns against simple cross-model rankings.

Why it matters

For developers, the gap between 78% and 29% is the key lesson: code that compiles and passes a demo can still mishandle failures, webhooks or subscription changes. Agent-written payment code needs human review against the full requirements.

For companies choosing coding agents, the results show large differences in both accuracy and cost per task, and that giving an agent official, current documentation through a plugin helps every model.

For API providers, Stripe’s approach is a template: publish realistic tests, then ship tools that steer agents to the right patterns.

What’s still open

  • The benchmark covers only Stripe integrations, so results say little about other kinds of coding work.
  • Strict scores are sensitive to grading errors, as Stripe itself notes.
  • Stripe says it will update the results with new models, cases and features over time.

Sources

  • Stripe developer blog, “Stripe evals: Measuring real-world coding agent performance,” 19 November 2026
  • Stripe evals results page
  • stripe/forge repository on GitHub

i10x Perspective

Benchmarks run by the companies whose APIs are being used are becoming more useful than generic coding scores, because they test what actually breaks in production. Stripe’s numbers are a reminder that “mostly right” is not good enough when money moves. More coding agent news in Coding.

Christopher Ort

Christopher Ort

Christopher Ort is the editor of i10X News at i10x.ai. He has worked hands-on with AI tools, models and agent workflows for years and leads i10X’s coverage of AI launches, funding and research. He is the author of “The Style Penalty”, an SSRN study on how frontier AI models judge AI-written CVs, covered by heise online and t3n.

Editor's picks

  1. Dragos canary study: attackers stole AI prompts and credentials at three AI inference providers
  2. Analysis: AI Agents in Banking Hit a Liability Wall
  3. CVE-2026-133631: FlowIntel chatbot let LLM call any API, exposed Ollama token in process table
  4. Stripe Evals: Coding Agents Score 78% on Payment Integrations, but Only 29% Pass Every Required Check
  5. Kong Volcano: A Self-Healing Agent Cloud Where Coding Agents Build, Host and Fix Apps