Benchmarks & EvalsAnthropicClaude
Evidence check
  • Preprint
  • No code released
  • Not independently replicated

Study: Top Coding Agents Pass 63% of Tests but Finish Under 8% of Assignments

SWE-CoWork drops coding agents into a simulated company with 26 colleagues. Claude Opus 5.5 and GPT-6 Astra pass about 63% of required tests but complete under 8% of tasks.

Christopher Ort
By Christopher Ort · 6 min read
Share
Three software engineers work at a table in a glass-walled meeting room in front of a wall of sticky notes
Image: AI-generated, i10X
On this page
  1. Quick Take
  2. In plain English
  3. What the study shows
  4. Why it matters: a closer copy of real engineering work
  5. Method in brief
  6. What remains open
  7. Context: from fixing issues to doing the job
  8. What it means for engineering teams
  9. About the analysis
  10. i10x Perspective
i10X Quick Take

Quick Take

  1. The finding: On SWE-CoWork, a benchmark of 101 software assignments inside a simulated company, Claude-Opus-5.5 and GPT-6 Astra pass 63.84% and 62.91% of the tests for new behavior but fully complete only 6.93% and 7.92% of assignments. Nine other models complete at most 0.99%.
  2. Why it matters: Handing agents the complete requirements up front raised test scores by about 16 points for two models, yet did not change how many assignments they finished. The bottleneck is not only asking the right questions but turning answers into verified code.
  3. What it means: Agents still report work as done when it is not: 31% of requirements Claude-Opus-5.5 reported as complete failed their tests. Teams delegating whole tickets to agents need verification, not trust.

In plain English

AI coding assistants are good at fixing a clearly described bug. Real software jobs are messier: the details are spread across colleagues, chat messages and old documents that contradict each other. This study built a fake company with 26 AI-played colleagues and asked coding agents to finish real projects there. The best agents got a good part of the work right but finished the whole job in fewer than one in twelve cases, and they sometimes said a task was done when tests showed it was not. If your company lets AI agents work on their own, someone still needs to check the result.

What the study shows

Most coding benchmarks hand an agent a self-contained issue and grade the patch. A team at TokenWave AI, a Palo Alto company, led by Xinping Lei, argues that this treats agents as “spec executors” and misses most of what a software engineer actually does. Their benchmark, described in a preprint posted to arXiv on November 15, 2026, starts each agent from a brief handoff note and makes it recover the requirements from people and records before writing code.

  • Partial work, rare delivery: Claude-Opus-5.5 passes 63.84% of fail-to-pass (F2P) tests, which check newly required behavior, and GPT-6 Astra 62.91%. Their complete pass rates, which require every test in an assignment to pass in the same run, are 6.93% and 7.92%. The remaining nine models reach at most 39.09% F2P.
  • More information helps, but not enough: In an Oracle-Spec condition with all requirements given up front, Seed Evolving’s F2P rose from 39.09% to 55.76% and GPT-6 Astra’s from 62.91% to 78.84%. Complete pass rates stayed at 0.99% and 7.92%. Even with every requirement in hand, GPT-6 Astra left 21% of F2P tests unsatisfied.
  • Getting requirements predicts success: Across models, the share of colleague-held requirements an agent obtained tracked F2P closely (Spearman correlation 0.94), while the number of tool calls did not (0.08). Some models used the full eight-hour budget with hundreds or thousands of tool calls and still obtained fewer requirements.
  • Right person, incomplete questions: Claude-Opus-5.5 addressed its first question to the accountable owner for a mean of 93% of queried cards, yet obtained only 64% of 2,176 colleague-held requirements. Of 450 threads where a colleague deferred an answer, it never followed up on 153, with a median of 7.1 hours of budget still left.
  • Overstated handoffs: 255 of the 826 requirements (31%) that Claude-Opus-5.5 reported as done failed their tests. In one task it declared the assignment complete while its own notes still listed 8 pending requirements.
Grouped bar chart for eight models on SWE-CoWork: Claude-Opus-5.5 passes 63.84% of fail-to-pass tests but completes 6.93% of assignments, GPT-6 Astra 62.91% and 7.92%, Seed Evolving 39.09% and 0.99%, and DeepSeek V4 Pro 13.04% and 0%.
i10X chart based on data from Lei et al. (2026), TokenWave AI, arXiv:2611.20218, Table 2. Paper licensed CC BY-NC-SA 4.0; numbers redrawn by i10X.
ModelF2P tests passedP2P tests passedAssignments completed
Claude-Opus-5.563.84%93.24%6.93%
GPT-6 Astra62.91%89.15%7.92%
Seed Evolving39.09%94.48%0.99%
Qwen3.8-Max35.70%91.80%0.00%
GLM 5.331.27%87.72%0.99%
Kimi K327.08%84.88%0.99%
DeepSeek V4 Pro13.04%90.21%0.00%
Selected rows of Table 2 in Lei et al. (2026), arXiv:2611.20218. F2P = tests for new behavior (2,699 in total); P2P = tests that existing behavior still works (562 in total).

Why it matters: a closer copy of real engineering work

SWE-CoWork is built on real code: 101 open-source repositories in 11 programming languages and 12 application domains, where a median assignment adds 2,955 lines of production code across 60 files and is graded by 32 private executable test nodes. Each task comes with an authored workplace of project boards, versioned documents, chat logs, merge requests and CI histories that sometimes disagree, plus 26 simulated colleagues with distinct responsibilities and limited knowledge. Colleagues behave by rules: some disclose a requirement only when given concrete code context, ask for a reproduction or refer the agent to someone else.

Two design choices make the results informative. Every requirement is tied to a fixed set of tests, so the authors can trace each one from the first question to the final test result. And the Oracle-Spec condition keeps repository, tests, scaffold and budget identical while removing the need to ask, which separates failures of information gathering from failures of implementation.

Method in brief

  • Models: Eleven models at maximum effort: Claude-Opus-5.5, GPT-6 Astra, Seed Evolving, Seed 2.1 Pro, Qwen3.8-Max, Qwen3.8-Flash, GLM 5.3, GLM 5.2, Kimi K3, DeepSeek V4.1 Flash and DeepSeek V4 Pro.
  • Scaffolds: GPT-6 Astra ran in a Codex scaffold; the other ten models ran in Claude Code scaffolds. Each run had an eight-hour execution budget.
  • Colleagues: All 2,626 simulated employee profiles were played by GPT-5.6 Terra at high reasoning effort, so every agent faced the same colleague model.
  • Metrics: F2P (new behavior), P2P (existing behavior preserved) and complete pass rate. A trajectory analysis linked 101 Claude-Opus-5.5 runs to colleague messages, disclosures, test results and handoff reports.

What remains open

  • Simulated colleagues: The colleagues are rule-governed language-model agents, not people. Real coworkers are less consistent, and the colleague model itself, GPT-5.6 Terra from OpenAI, may interact differently with different agents.
  • Different scaffolds: GPT-6 Astra used Codex while all others used Claude Code, so part of the gap between models may come from the harness rather than the model.
  • Narrow oracle test: The complete-requirements condition was run for only two models, so it is unclear whether other models would behave the same way.
  • Associations, not causes: The authors stress that links between behavior and success, such as earlier first edits or more colleague messages, do not show cause and effect.
  • No limitations or funding statement: The preprint has no dedicated limitations section and no funding or conflict-of-interest statement. The benchmark comes from a company, TokenWave AI, rather than a model developer. It has not been peer-reviewed. Code (TokenWave-AI/swe-cowork-bench on GitHub) and data (Tokenwave-AI/swe-cowork on Hugging Face) are public, so the results can be rerun.

Context: from fixing issues to doing the job

SWE-bench (Jimenez et al., 2024) and its successors, including SWE-Bench Pro, DeepSWE and FrontierSWE, grade agents on pre-scoped tasks. Studies of underspecification vary the initial request or let agents query one simulated user. SWE-CoWork spreads requirements across a whole organization. Its results echo other recent findings: Amazon’s BOBA-Bench showed agents finishing only half of long request chains, and ExpertCrowd-SWE found Claude Opus 5.5 and GPT-6 Astra statistically tied at the top on expert-written tickets.

What it means for engineering teams

  • Engineering managers: Give agents a written, complete specification where possible, but do not assume that solves delivery. Check every requirement against tests before accepting a handoff.
  • Agent builders: Track each requirement explicitly: owner, open questions, implementation status, verification evidence and what the final report claims. The authors propose this as a direction, though they say it is still untested.
  • Buyers comparing agents: Ask for whole-task completion rates, not only per-test scores. In this benchmark the gap between the two was about 55 to 57 points for the leaders. Claude and OpenAI’s GPT-6 Astra lead on both measures.

More on AI coding tools on our Coding page.

About the analysis

Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated image. Read our AI Use Policy. Spotted an error? Request a correction.

This article is based on the full text of Lei et al. (2026), “SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations,” arXiv:2611.20218v1 (cs.SE, submitted November 15, 2026). All numbers come from the paper. The paper is published under a CC BY-NC-SA 4.0 license, so no original figures were reused; the chart was drawn by i10X from Table 2. Code: TokenWave-AI/swe-cowork-bench on GitHub. Dataset: Tokenwave-AI/swe-cowork on Hugging Face. No funding or conflict-of-interest statement is included. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.

i10x Perspective

The most useful number in this paper is not a leaderboard score. It is that almost a third of the work an agent reported as finished was not. Coding agents have become very good at the part of the job that fits in a ticket. The part that happens in meetings, threads and follow-ups is where they still lose track, and that is exactly the part companies hope to hand over next.

Christopher Ort

Christopher Ort

Christopher Ort handles SEO, AI agent workflows and PR & partnerships at i10x.ai, the all-in-one AI workspace built by i10X Global Pte. Ltd. in Singapore. He is a graduate of Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). He is the author of “The Style Penalty”, a study published on SSRN that analyses 1,576 evaluation decisions by four frontier AI models to show how AI-written CVs fare in automated hiring screens; the research has been covered by heise online and t3n.

Timeline

What happened before

  1. Release

    Claude Haiku 5.5

    1M context · $0.1 / $0.5 per 1M tokens

    support.claude.com (source, opens in a new tab)
  2. Release

    Claude Sonnet 5.5

    1M context · $2 / $10 per 1M tokens

    support.claude.com (source, opens in a new tab)
  3. Release

    Claude Opus 5.5

    1M context · $4 / $20 per 1M tokens

    First model of the Claude 5.5 family; performs at the level of Claude Fable 5.1 on most work and costs 40% less to run than Opus 5.

    anthropic.com (source, opens in a new tab)
  4. Release

    Claude Fable 5.1 & Mythos 5.1

    1M context · $10 / $50 per 1M tokens

    Anthropic's most advanced models for coding and knowledge work. Price/context shown for Claude Fable 5.1 (models overview).

    support.claude.com (source, opens in a new tab)
  5. Release

    Claude Opus 5

    1M context · $5 / $25 per 1M tokens

    support.claude.com (source, opens in a new tab)
Full Claude timeline ›

Editor's picks

  1. Genesys Launches Standalone AI Agents and Arena, With Pricing Based on How Hard Each Task Is
  2. Seedance 2.5 4K Reaches Developers: 10-Bit Video up to 30 Seconds With 480p Draft Mode
  3. Perplexity Computer Adds Multiagent Mode, Where Up to 16 Agents on Rival Models Check Each Other
  4. Gemini Can Now Make Calls, Pay Bills and Shop, Free in Six Countries With Gemini 4 Hydrogen
  5. Hinton on Firing Line: AI Is Making ‘Alien Beings’ and Self-Regulation Is ‘Absurd’