Which AI coding agent is best? For most of 2026, that question was answered with a model name. A cluster of new tests now suggests the answer depends just as much on the harness — the software around the model that decides what it reads, which tools it may call, how it remembers progress and when it checks its own work. On November 8, Y Combinator CEO Garry Tan posted early results from his own open-source harness evaluations. Builders then paired them with two other data sets: a 28,849-trial Harness Leaderboard and an 18.4-point spread for a single Claude model on Terminal-Bench 2.0. The picture is more nuanced than “harness beats model,” and the price tag turns out to be the clearest signal.
Quick Take
- Tan’s test: Seven setups — Claude Code, OpenClaw, Hermes and QM on Opus, plus Codex, OpenClaw and Hermes on Sol — passed 71–79% of the same 20 coding tasks. Tan says 14 of the 20 tasks scored identically everywhere, so “the test was too easy to separate them.”
- Cost is the real spread: On Opus, OpenClaw paid about $0.44 per solved task versus $0.22 for Claude Code, $0.24 for Hermes and $0.19 for QM. On Sol, Codex was cheapest and fastest at $0.17.
- Bigger data: Joël Niklaus’s Harness Leaderboard ran 28,849 trials ($4,746 of inference) across nine harnesses; average scores ranged from 27.3% to 51.4% with the model held fixed.
- The 18.4-point gap: On the public Terminal-Bench 2.0 board, Claude Opus 4.6 scores 76.4% in Stanford’s Meta-Harness and 58.0% in Claude Code — but only 4.9 points apart when you keep verified runs only.
What happened
On November 8, Tan wrote that “out of curiosity” he was “doing some open source eval work on harnesses with frontier models,” and shared a first summary. He published the project as g-harness-evals on GitHub, describing it as “open source harness evals: which agent harness does tasks the best, for the least cost?”
His summary makes two points. First, “which harness is smarter? We can’t tell yet.” Seven setups were given the same 20 coding tasks and “all of them passed 71–79% of the time.” Because 14 of the 20 tasks scored the same in every setup, Tan called it “a test problem, not a tie,” and said a harder rerun on Terminal-Bench is running. Second, “what it costs differs a lot.” On Opus, OpenClaw paid about $0.44 per solved task, Claude Code $0.22, Hermes $0.24 and QM $0.19; OpenClaw was also slowest at about 99 seconds per try against 40 for Claude Code. On Sol, Codex was the cheapest and fastest at $0.17. A second, separate run found the same pattern, he wrote, concluding: “For now, choose on cost and speed: Claude Code on Opus, Codex on Sol.”
Within hours, the “harness vs. model” framing spread across X, combining Tan’s chart with two other sources that had been circulating since earlier in the week.
Background: from prompts to harness engineering
“Harness” has become the 2026 word for everything an agent product adds to a raw model: the system prompt, tool definitions, the agent loop, context compaction, memory files, permission rules, sub-agents and verification steps. A July 2026 source-code study of eleven production coding agents, “Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents” (arXiv:2609.00006), mapped them to seven common subsystems: agent loop, LLM integration, tools and actions, memory and context, safety and permissions, orchestration, and extensibility.
i10x has followed the theme for weeks: ByteDance Seed’s HarnessDev study found GPT-5 solving 35.2% of Terminal-Bench 2.1 in one harness and 49.6% in another, covered in LLMs Can Build Their Own Agent Harnesses; Anthropic’s guidance on a single agent loop is in Anthropic’s single-agent-loop guidance; and community setups after Boris Cherny’s playbook are in Developers Share Claude Code Agent Tutorials After Cherny’s Playbook. What is new this week is measurement at scale, plus a well-known investor running his own evals in public.
How the three data sets compare
Harness Leaderboard. Released on November 5 as a Hugging Face Space, Joël Niklaus’s board holds model, provider and reasoning effort fixed and swaps only the harness: Claude Code, Codex, Pi, OpenCode, Hermes Agent, mini-swe-agent, Kimi Code, Qwen Code and OpenClaw. It runs Terminal-Bench 2.1 and DeepSWE with three open models — GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint — and three attempts per task. According to AI Tracker and a detailed write-up of the board, the bash-only mini-swe-agent averages 51.4%, OpenClaw 27.3%, and Kimi Code, Hermes Agent, Pi and Claude Code are statistically tied with the leader. Cost per solved task ranges from $0.29 (Pi) to $3.35 (OpenCode), and the best harness changes by model.
Terminal-Bench 2.0. On the public board, Claude Opus 4.6 appears with nine harnesses. Stanford IRIS Lab’s Meta-Harness — an automatically evolved scaffold built on Terminus-KIRA that injects an environment snapshot into the first prompt — scores 76.4%; Claude Code scores 58.0%. That is the 18.4-point gap now circulating. But an analysis of the board by developer Quang Lee notes that “with only verified rows, Opus 4.6’s spread falls from 18.4 to 4.9 points,” and Terminal-Bench 2.1 later fixed 28 of the 89 tasks; Claude Code with Opus 4.6 rose to 70.1% on 2.1, according to a summary of the release notes.
Tan’s evals. Small (20 tasks), frontier-model only, and explicitly preliminary — but the only one of the three that compares commercial harnesses on their makers’ own flagship models.
| Test | What was held fixed | What changed | Headline result | Caveat |
|---|---|---|---|---|
| Garry Tan, g-harness-evals (Nov 8) | Model (Opus or Sol) and 20 coding tasks | 7 harness setups | All passed 71–79%; 14 of 20 tasks scored identically; Opus cost per solved task $0.19–$0.44 | Tan: the test was “too easy” — a Terminal-Bench rerun is running |
| Joël Niklaus, Harness Leaderboard v0.1 (Nov 5) | Model, provider, reasoning effort; 3 attempts per task | 9 harnesses on 3 open models | Average 27.3% (OpenClaw) to 51.4% (mini-swe-agent); five harnesses statistically tied for first | Open Flash-tier models only; single operator, not yet replicated |
| Terminal-Bench 2.0 public board | Claude Opus 4.6 | 9 harness submissions | 76.4% (Stanford Meta-Harness) vs. 58.0% (Claude Code) — 18.4 points | Public submissions differ in budgets and prompts; verified rows only: 4.9 points |
Pros and cons
For users
- Pro: If you pay per token, the harness can halve your bill at the same pass rate — Tan’s Opus numbers range from $0.19 to $0.44 per solved task.
- Con: Pass rates on easy tasks barely move, so switching tools for “smarter” output may disappoint; differences only show on hard, long-horizon work.
For developers
- Pro: Context handling, environment discovery and verification are things you can change yourself — Meta-Harness’s main trick is saving “2-5 early exploration turns” by describing the sandbox up front.
- Con: Results are model-specific. The Harness Leaderboard crowns a different harness for each open model, and a configuration tuned on one benchmark may not generalize to your repository.
For companies
- Pro: Tooling decisions can be made on measured cost and latency, not brand loyalty, and open boards publish every trajectory for audit.
- Con: Benchmarks still disagree on how big the harness effect is (from roughly 5 to 24 points depending on the data set), so vendor claims of a harness “advantage” need in-house testing.
What X is saying
Out of curiosity I am doing some open source eval work on harnesses with frontier models and here’s what is coming back so far
— Garry Tan (@garrytan) November 8, 2026 · View on X
Several replies read the result as a ceiling effect rather than a draw. Ibrahim Denis Fofanah summed it up: “Seven coding-agent setups. Same 20 tasks. All passed 71–79% of the time. That isn’t a tie. It’s a test too easy to rank them.”
@garrytan The useful line is not “they’re tied.” 14 of the 20 tasks scored the same in every setup, so this test cannot rank harnesses yet — while cost already can: about $0.44 vs $0.22 per solved task on Opus, and Codex cheapest on Sol. Waiting on the harder Terminal-Bench rerun.
— puppyone (@puppyone_ai) November 9, 2026 · View on X
Same open models, swap only the harness, and tasks solved go from 27.3% to 51.4%. That gap is from a new Harness Leaderboard. 28,849 trials and $4,746 of inference across nine coding-agent harnesses, on Terminal-Bench 2.1 and DeepSWE, with GLM-5.3-Flash, DeepSeek-V4.1-Flash and …
— Mr. Ånand (@Astrodevil_) November 8, 2026 · View on X
Cici (@LabubuNoBB) added a caution worth keeping: the 18.4-point Terminal-Bench gap “is evidence about specific configurations, not a universal ranking rule.”
What Grok says
Asked to summarize Tan’s chart, Grok replied (in Chinese) that the seven configurations performed closely at 71–79% on 20 coding tasks, that the test was too easy to tell them apart, and that a harder Terminal-Bench rerun is underway — consistent with Tan’s own wording.
Critical analysis
The viral version of this story — “harness matters more than the model” — overreaches. None of the three data sets swaps models and harnesses in one controlled experiment, so none can say which matters more. What they do show is narrower and more useful: with the model held fixed, harness choice moves scores anywhere from a few points (Tan’s easy tasks, verified Terminal-Bench rows) to more than 20 (Harness Leaderboard on open Flash-tier models), and moves cost and latency far more reliably than accuracy.
That asymmetry matters. Weaker or cheaper models seem to benefit most from a good harness, which fits the Harness Leaderboard’s open-model results and the older Haiku numbers in the Meta-Harness paper. Frontier models on easy tasks converge. The 18.4-point headline mixes both effects with differences in budgets and prompts across public submissions — and Claude Code’s 58.0% partly reflected broken tasks that Terminal-Bench 2.1 later fixed.
Tan’s evals deserve credit for publishing method and code, but 20 tasks is a smoke test. The upcoming Terminal-Bench rerun will be the more meaningful result.
What’s still open
- Will Tan’s harder Terminal-Bench rerun separate the seven setups — and does the cost ranking hold?
- How do Claude Code and Codex perform on the Harness Leaderboard with their makers’ own models, which the board does not currently test?
- Can harness gains found by automatic search, like Meta-Harness, transfer from benchmarks to real repositories?
- Will vendors start publishing harness-level cost-per-solved-task figures alongside model benchmarks?
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This independent, research-based i10x analysis draws on Garry Tan’s posts and GitHub repository of November 8, 2026, Joël Niklaus’s Harness Leaderboard (version 0.1, November 5) as reported by AI Tracker and Pasquale Pillitteri, the Meta-Harness paper and artifact from Stanford IRIS Lab, public Terminal-Bench 2.0 results, and arXiv:2609.00006. Figures are as reported by those sources and were not independently re-run. More on Anthropic, Claude and OpenAI.
- GitHub: garrytan/g-harness-evals
- Garry Tan on X: early harness eval results
- Hugging Face: Harness Leaderboard (Joël Niklaus)
- AI Tracker: Harness Leaderboard compares nine coding-agent harnesses
- Pasquale Pillitteri: Harness Leaderboard reveals a 24-point swing
- Stanford IRIS Lab: Meta-Harness, 76.4% on Terminal-Bench 2.0
- Meta-Harness: End-to-End Optimization of Model Harnesses (paper)
- Quang Lee: The best agent harness is a while loop and a lot of upkeep
- arXiv:2609.00006 — Harness Engineering: A Source-Code Study of Eleven Systems
i10x Perspective
The harness debate is really a pricing debate in disguise. At the frontier, on everyday tasks, today’s best harnesses produce similar answers — but one can cost twice as much and take twice as long as another. For teams buying coding agents by the seat or by the token, that is the number to track. The more interesting long-term signal is Meta-Harness: if harnesses can be evolved automatically against a benchmark, the “secret sauce” of agent products will commoditize quickly, and differentiation will move to what no benchmark captures — permissions, audit trails, integration with a company’s own repositories, and how gracefully an agent fails.
