Quick Take
- The finding: Ten coding agents complete about 95% of programming requests one at a time, but only 51% of chains of 20 requests that build on each other, according to Amazon’s new BOBA-Bench.
- The cause: Agents declare work done once their own tests pass, miss mistakes, and then build the next request on broken code, which then fails about twice as often.
- The fix: A single external “verification failed” signal plus one repair round raises chain completion from 51% to 59% on average, but also adds regressions and extra calls.
Coding agents are increasingly asked to carry out whole sequences of changes on their own, each one resting on the last. Most benchmarks still test them on one isolated task, starting from code written by humans and graded once. A new preprint from Amazon, “Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone” by Alejandro Cuadron and Arpit Gupta, tests what happens when agents are left to work through long chains of requests without outside checks. The short answer: their own missed mistakes pile up.
What the study shows
BOBA-Bench contains 280 chains of 20 consecutive requests drawn from the commit histories of 140 real open-source Python projects, 5,600 requests in total. Each of ten agents ran every chain eight times. A chain only counts as completed if every request passes the hidden tests.
- Chains break agents: Average chain completion (pass@1) is 51%. The best agent, Claude Opus 5.5, reaches 66.7%, the weakest, Claude Opus 4.8, 39.2%. Within the chains, agents still complete 66 to 84% of the individual requests.
- Not a quality problem: Once agent code has been verified, it is nearly as good a base for later work as human code. The gap between building on verified agent code and on the developers’ own code ranges from −3.6 to +0.8 points.
- Missed mistakes compound: A request that starts from broken code fails about twice as often, 9 to 19 points more than from the developers’ state. A silent miss raises the failure rate of the next request by 18 points on average, and only 14 to 33% of failing tests are ever repaired later. Regressions occur in 23 to 43% of attempts.
- Review does not save them: Asked whether an implementation is correct, agents are right 53% of the time. Lenient reviewers (the four Claude models, GLM-5.3, GLM-5.3-Flash and Kimi K3) accept 77 to 95% of wrong implementations; strict ones (GPT-6 Astra, GPT-6.1, GPT-5.6) reject 41 to 58% of correct code.
- One bit helps: A plain “verification failed” message and one repair round lift average chain completion from 51% to 59%, with gains from 1.1 to 22.1 points; seven of ten confidence intervals exclude zero.
| Agent | Chain pass@1 (8 attempts) | Autonomous (1 attempt) | With verification signal |
|---|---|---|---|
| Claude Opus 5.5 | 66.7% | 66.8% | 73.9% |
| Claude Opus 5 | 61.6% | 60.7% | 69.6% |
| GPT-6 Astra | 53.2% | 51.1% | 53.9% |
| Claude Fable 5 | 51.5% | 51.8% | 63.9% |
| GLM-5.3 | 51.2% | 46.4% | 68.6% |
| GLM-5.3-Flash | 51.1% | 51.1% | 67.1% |
| GPT-6.1 Sol | 50.3% | 48.9% | 52.9% |
| GPT-5.6 Sol | 45.2% | 45.7% | 46.8% |
| Kimi K3 | 44.2% | 45.4% | 52.5% |
| Claude Opus 4.8 | 39.2% | 38.2% | 44.6% |
Why it matters: Verification, not raw skill, sets the ceiling
The study separates two explanations that are usually mixed up. If agents simply wrote worse code, building on verified agent code would hurt. It barely does. What hurts is unverified code: agents trust their own tests, and those tests miss errors that later requests inherit. That makes the bottleneck measurable and fixable, and it explains why single-task scores near the ceiling say little about autonomous multi-step work.
The design is careful. Chains come from real commit histories rather than synthetic tasks, every chain was run eight times, results come with repository-clustered bootstrap intervals, and the authors tested whether the effect depends on the harness (Codex CLI, OpenHands SDK and pi do not change the picture) or on memorization (it does not explain the results).
Method in brief
- Benchmark: 280 chains of 20 requests from 140 Python repositories; each request is a model-written specification of a real change, graded by hidden tests that also check earlier requests for regressions.
- Agents: Claude Opus 5.5, Opus 5, Opus 4.8, Claude Fable 5, GPT-6 Astra, GPT-6.1 Sol, GPT-5.6 Sol, GLM-5.3, GLM-5.3-Flash and Kimi K3, all run in mini-swe-agent 2.4.6.
- Comparisons: Requests from the developers’ code vs. from the agent’s own code, verified vs. unverified foundations, a review task with 2,180 items per model, and verification arms that send a failure signal with or without naming the failing test.
What remains open
- Narrow scope: The benchmark is Python-only, correctness is defined by fixed tests, and many of the repositories are small.
- Unreviewed specifications: The request texts were written by a model and not checked by humans.
- Contamination risk: The code changes predate the models’ training cutoffs, and three Claude models reproduce 14 to 19% of sampled change blocks near-verbatim. The authors find memorization does not explain the results, but it remains a caveat.
- Repair has a price: The repair round raises the regression rate by 3.2 to 8.9 points and adds 47 to 294 model calls per chain.
- Not yet reproducible: The paper says the benchmark, data, trajectories and grader are released on GitHub, but the linked repository was not publicly reachable on November 3, 2026. The study is a preprint without peer review, and no funding or conflict-of-interest statement is given; both authors work at Amazon.
Context: From single tasks to long-horizon agents
Benchmarks such as SWE-bench made single-issue repair the standard yardstick for coding agents. BOBA adds a long-horizon test that fits how tools from Anthropic, OpenAI and Moonshot are now marketed: as autonomous developers. It also lines up with earlier findings on i10X, such as coding agents that comply too readily during refactoring and agents that launder approvals during context compaction. More research is collected in Agent Research and Coding.
What it means for teams using coding agents
- Keep a test gate in the loop: Run your own test suite after every agent step, not only at the end, and feed failures back to the agent.
- Be specific: Naming the failing test once was worth about three rounds of bare rejection in the study.
- Don’t use agents as reviewers of agent code: With 53% accuracy, a second agent is no substitute for tests or human review.
- Read scores carefully: A 95% single-task score does not mean an agent can run a 20-step backlog unattended.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint (arXiv:2611.02207v1, submitted November 1, 2026), including the appendix. All numbers come from the paper; i10X did not rerun the benchmark. The chart was drawn by i10X from Table 41; the paper is licensed CC BY 4.0. AI tools assisted with research and drafting; the selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
BOBA shifts the question from “how smart is the agent?” to “what checks can it reach?”. For anyone handing Claude, GPT or Kimi a backlog, the practical lesson is simple: the agent is only as reliable as the tests you give it.
