Quick Take
- The finding: On 100 coding tasks written by practising engineers for real open-source codebases, Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) lead a field of seven frontier models, and the gap between them is not statistically significant.
- Why it stands out: Unlike crowded leaderboards, the benchmark spreads models over 35 points in four distinct tiers. No task was fully solved by every model, and full-credit runs are rare: 26.6% for the leader, 1.0% for Qwen 3.8 Max and Grok 4.6.
- What it means: The two leaders are complementary rather than ranked: Opus 5.5 solves more tasks at least once, GPT-6 Astra is more consistent and needs about a fifth of the steps. Pick by workflow, not by one number.
Coding agents now get hired to change real codebases, yet the benchmarks used to compare them are fraying. OpenAI stopped reporting SWE-bench Verified in February 2026 after an audit found problems in 59.4% of 138 checked tasks, and on the authored DeepSWE v1.1 leaderboard four frontier models sit within 0.9 points of each other. A new technical report from Toptal ExpertCrowd, “ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases” by Josip Vrdoljak and six co-authors, tries a different recipe: tickets written by engineers who know the codebase, graded the way a maintainer reviews a pull request.
What the study shows
- Clear tiers: Claude Opus 5.5 scores 67.3% and GPT-6 Astra 63.7%, followed by Claude Fable 5.1 (46.5%) and Gemini 3.8 Flash (40.0%). Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 form a statistically indistinguishable group at 32.2–33.5%. The seven-model mean is 45.1%.
- The top two are tied: Opus 5.5 leads GPT-6 Astra by 3.6 points, but the paired 95% interval runs from −1.4 to +9.7 (p = 0.20). Each wins many tasks: 53 to 41, with 6 ties. Every other model trails Opus 5.5 significantly (p < 0.001).
- Lots of headroom: On 80 of the 100 tasks no model earns full credit on all five runs. The hardest task, a SystemVerilog export for the CirC compiler, left all seven models between 2.3% and 2.7%.
- Breadth versus reliability: Opus 5.5 reaches full credit at least once on 47 tasks versus 36 for GPT-6 Astra, but GPT-6 Astra hits full credit on all five runs for 15 tasks versus 9, and its scores vary less within a task (SD 7.3 vs. 13.6 points).
- Effort is not capability: GPT-6 Astra needs a median of 33 agent steps per run and reaches 95% of Opus 5.5’s score with about a fifth of its steps (Opus 5.5: 160). Gemini 3.8 Flash takes the most steps (166) and ranks fourth.
- Maintainer rules bite: 7.8% of all runs were capped by a failed must-pass criterion and 2.3% zeroed by a failed gate such as “leave the existing tests untouched”. The strongest models lost the most, but the ranking did not change.
| Model | Score (95% CI) | Full-credit runs | Median steps per run | Median container time |
|---|---|---|---|---|
| Claude Opus 5.5 | 67.3 ± 4.7 | 26.6% | 160 | 80 min |
| GPT-6 Astra | 63.7 ± 5.5 | 23.4% | 33 | 36 min |
| Claude Fable 5.1 | 46.5 ± 3.7 | 8.6% | 127 | 56 min |
| Gemini 3.8 Flash | 40.0 ± 4.3 | 3.8% | 166 | 40 min |
| Qwen 3.8 Max | 33.5 ± 3.3 | 1.0% | 137 | 100 min |
| DeepSeek V4.1 Flash | 32.5 ± 3.7 | 3.0% | 160 | 51 min |
| Grok 4.6 | 32.2 ± 3.5 | 1.0% | 73 | 69 min |
Why it matters: grading like a maintainer
The design targets the two known weak spots of mined benchmarks. Contamination: tasks are new changes that were never published as fixes, 76 of 100 are held out, and every task was compared with 499,647 records from 93 public benchmark datasets without finding a duplicate. Grading: instead of inheriting the tests of one historical patch, each task has a weighted rubric written by its author, with mostly executable checks through public interfaces. Each task image only builds if the reference solution scores exactly 1.0 and both the untouched repo and a comment-only change score 0. Nearly half the tasks also ship alternative, lazy and “cheat” patches the grader must classify correctly, and LLM judges carry just 0.85% of the total weight. The statistics are careful too: paired permutation tests with Holm correction, and both task-sampling and run-to-run intervals. The former are two to ten times wider, which the authors rightly point out most leaderboards understate.
Method in brief
- Tasks: 100 multi-file change requests across 87 open-source projects and 11 languages (Go leads with 36). Median ticket: 148 words; median reference solution: 471 changed lines across 4.5 files. 82 base commits date from August or September 2026.
- Authors: 39 engineers from Toptal’s network contributed; the final tasks come from 17 of them. Each task was quality-checked by a senior engineer.
- Models and harness: Seven models, each run through the HUD platform with its provider’s native tool interface at default high reasoning effort, with no system prompt and no vendor product such as Claude Code or Codex CLI.
- Runs: Five runs per model and task with a 1,000-step limit, 3,495 scored runs collected from October 1 to 10, 2026. Runs that hit step or token limits were rerun (27 for Opus 5.5, 55 for Gemini 3.8 Flash, 2 for Grok 4.6).
What remains open
- Harness differences: Each provider gets its own tool interface, so part of a gap may reflect tool design rather than the model. A shared single-tool rerun is listed as future work.
- One reasoning setting: Each model ran at its platform’s default high effort. Effort was not swept, so other settings could change both scores and step counts.
- Sample size: With 100 tasks, the authors say differences under about 5 points between neighbouring models should not be over-interpreted.
- Public code underneath: The repositories are public and likely in training data, so EC-SWE tests problem solving in familiar-looking code.
- Who measures: This is a vendor technical report, not peer reviewed. The authors work for Toptal ExpertCrowd, which runs the leaderboard and supplies expert talent; no funding or competing-interest statement is given. Only 24 tasks are public, so outsiders can verify part of the measurement.
Context: after the SWE-bench era
SWE-bench mined 2,294 GitHub fixes and became the standard; later variants added languages, multimodal tasks and private splits. Research showed models reproduce SWE-bench gold patches far more often than comparable unseen code, and OpenAI’s own audits in 2026 pushed the field toward authored tasks. DeepSWE (113 original tasks, one bash tool for all models) is the closest neighbour. Both benchmarks agree on the top two but diverge below, most clearly for DeepSeek V4.1 Flash: third on DeepSWE, sixth here. The authors call that comparison indicative, not controlled. We covered similar harness effects in harness versus model in coding benchmarks and real-world agent requests in BOBA-Bench.
What it means for engineering teams
- Match the model to the workflow: If you can run several attempts and verify them, Opus 5.5’s breadth helps. If one attempt has to land, GPT-6 Astra’s consistency and lower step count matter more.
- Budget for effort: Step counts varied five-fold and container time from 36 to 100 minutes, with little link to quality. Track cost and latency next to accuracy.
- Guard your tests: Gates caught runs that edited tests or broke stated constraints. Make “existing tests untouched” a hard check in your own agent pipelines.
- Expect partial work: Most strong runs earned at least 40% of the credit but few were merge-ready. Plan human review for multi-file changes.
More on the Agents research page and the Claude hub.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of Vrdoljak, Singh, Steunebrink, Yarotsky, Lorenzo, Nicoletti and Pigott (2026), “ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases,” arXiv:2611.13246v1 (cs.AI, technical report, submitted November 9, 2026), including the appendix. All numbers come from the paper. The chart was redrawn by i10X from Table 3; the paper is distributed under the arXiv non-exclusive license, so no original figures were reused. No funding or competing-interest statement is included; the authors are affiliated with Toptal ExpertCrowd. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
The headline is less “Claude beats GPT” than “benchmarks can still separate frontier agents if tasks look like real tickets and grading looks like real review.” The caveat is obvious: a talent marketplace grading AI coders has its own interests, and 76 hidden tasks ask for trust. Still, the design choices, from calibrated graders to honest uncertainty intervals, are ones every coding leaderboard should copy.
