Stripe has published Stripe evals, a public benchmark that measures how well AI coding agents build real payment integrations. Across 14 combinations of models and coding tools, the agents scored 78.1% on average. But when Stripe counted only runs that met every required criterion, the rate fell to 29.4%.
Quick Take
- What’s new: Stripe evals and an open eval framework, published on Stripe’s developer blog on 19 November 2026.
- The test: 69 scenarios, 24 one-time payments, 26 subscriptions and 19 Connect marketplace cases, each run twice with and without Stripe’s agent plugin.
- Headline gap: the average score across all agents falls from 78.1% to 29.4% under the strict pass rate.
- Top strict scores: GPT-6 Astra in Codex reached 52.9% with the plugin, GPT-6.1 Sol 50.7%, and Fable 5.2 in Claude Code 43.2%.
- Plugin effect: all 14 agents scored higher with the plugin, by about 3 to 8 points, and web searches fell 96%.
In plain English
Stripe handles online payments for many companies, and more developers now ask AI assistants to write the code that connects their apps to it. Stripe tested how good these assistants really are. Most of their code looks mostly right, but only about three in ten attempts got every essential detail correct, and with payments a small mistake can mean money goes to the wrong place.
What happened
Stripe researchers Kiran Prasad, Yiing Chau Mak and Manish Kurse explained why they built the benchmark: agents “bias towards older APIs that are more prevalent in their pretraining corpus, miss key features, and hallucinate.” Existing coding benchmarks, they wrote, “check whether the code passes deterministic checks, not whether the API integration is correct.”
Stripe tested Claude Code with Haiku 5.5, Sonnet 5.5, Opus 5.5, Fable 5.1 and Fable 5.2, Codex with GPT-6 Luna, GPT-6.1 Sol and GPT-6 Astra, and OpenCode with Gemini 4 Argon, Gemini 3.8 Flash, Grok 4.7, Kimi K3, GLM 5.3 and MiniMax M3.1 Flash. That is 276 runs per agent.
Background
Vendors are increasingly writing their own tests for coding agents, because general benchmarks miss product-specific mistakes. Supabase did this for databases with Supabase Evals, and recent tests show that the harness around a model can matter as much as the model.
How it works
- A simulated developer: a roleplaying agent starts with a short request, keeps a private list of two to four needs, and asks follow-up questions one at a time, like a real client.
- Real workspace: coding agents could inspect the repository, write files, run commands and revise their work over several turns.
- Grading: judges score each rubric criterion separately using the conversation and a snapshot of the code. Stripe used GPT-6.1 Sol as roleplayer and judge and ran a check with Opus as judge, which “showed no bias.”
- Two scores: the integration success rate averages all required and recommended criteria; the strict pass rate needs every required criterion to pass.
What the results show
- Best averages: with the plugin, GPT-6 Astra scored 90.0% and GPT-6.1 Sol 89.6%; Opus 5.5 reached 86.2%.
- Cost differences: a run cost about $0.07 with GPT-6 Luna and $3.63 with GPT-6 Astra when using the plugin.
- Fewer outdated calls: the plugin cut outdated parameter references from 7% to 1.3% of parameters in API calls.
- Category swings: agents that led on payments and subscriptions did not lead on Connect, so Stripe warns against simple cross-model rankings.
Why it matters
For developers, the gap between 78% and 29% is the key lesson: code that compiles and passes a demo can still mishandle failures, webhooks or subscription changes. Agent-written payment code needs human review against the full requirements.
For companies choosing coding agents, the results show large differences in both accuracy and cost per task, and that giving an agent official, current documentation through a plugin helps every model.
For API providers, Stripe’s approach is a template: publish realistic tests, then ship tools that steer agents to the right patterns.
What’s still open
- The benchmark covers only Stripe integrations, so results say little about other kinds of coding work.
- Strict scores are sensitive to grading errors, as Stripe itself notes.
- Stripe says it will update the results with new models, cases and features over time.
Sources
- Stripe developer blog, “Stripe evals: Measuring real-world coding agent performance,” 19 November 2026
- Stripe evals results page
- stripe/forge repository on GitHub
i10x Perspective
Benchmarks run by the companies whose APIs are being used are becoming more useful than generic coding scores, because they test what actually breaks in production. Stripe’s numbers are a reminder that “mostly right” is not good enough when money moves. More coding agent news in Coding.



