Benchmarks & Evals
Evidence check
  • Preprint
  • No code released
  • Not independently replicated

Multi Agent Coordination Bench: Eight AI Agents Beat One, but Not Because They Work Together

Bespoke Labs' Multi Agent Coordination Bench tests AI agent teams on twelve four-hour tasks. Eight agents beat one (0.42 vs 0.35), but eight agents that could not talk scored even higher (0.44). Only Opus 5.5 gained from messaging, and teams cost 8.8 times as much.

Christopher Ort
By Christopher Ort · 8 min read
Share
An empty rowing boat with its oars out of sync rests beside a wooden dock on a misty lake at dawn.
Image: AI-generated, i10X
On this page
  1. Quick Take
  2. In plain English
  3. What happened
  4. Background: why agent teams are suddenly a big deal
  5. How it works
  6. What the numbers say
  7. The failures, in the agents' own logs
  8. Pros and cons
  9. What X is saying
  10. Critical analysis
  11. What's still open
  12. About the analysis
  13. i10x Perspective

More AI agents should mean more done, right? A new benchmark from Bespoke Labs suggests that today’s agent teams behave less like a well-drilled crew and more like eight people editing the same document without talking. Multi Agent Coordination Bench (MAC Bench), released on November 18, tests teams of frontier AI agents on twelve long, interdependent tasks, from designing a Mars sample-return mission to restoring a national power grid. Its headline finding: a team of eight agents beats a single agent, but almost entirely because it has more workers, not because the agents coordinate. Here is what the benchmark measures, what it found and why it matters for anyone building agent swarms.

i10X Quick Take

Quick Take

  1. What: Bespoke Labs released Multi Agent Coordination Bench, twelve four-hour tasks built so that one agent’s work depends on its teammates’ work.
  2. Main result: Averaged over three models, a single agent scored 0.35, a team of eight that could message scored 0.42, and eight agents that could not talk scored 0.44.
  3. Model gap: Opus 5.5 was the only model whose teams gained from messaging, reaching 0.53 against 0.41 for GPT-6.1 Sol and 0.32 for Gemini 4 Argon.
  4. Cost: A team of eight cost 8.8 times as much as one agent at list prices, and a single agent delivered the most score per dollar in every task and model pairing.
  5. The catch: These are Bespoke Labs’ own results on three models, each in its developer’s harness, and they have not been independently replicated.

In plain English

Many AI companies are betting that groups of AI assistants working together can solve bigger problems than one assistant alone. This test checked whether those groups actually cooperate. It found that eight assistants do get more done than one, but mostly because there are more of them, not because they help each other. In fact, letting them chat often made things slightly worse, and running eight costs almost nine times as much.

What happened

Bespoke Labs, a startup that builds training and evaluation environments for AI agents, published the benchmark with a long technical report and a launch thread on X:

Co-founder Alex Dimakis, a professor at UC Berkeley, put it more bluntly:

Background: why agent teams are suddenly a big deal

Adding more agents is emerging as a new way to throw compute at hard problems. The report opens by noting that OpenAI deployed approximately 10,000 agents on a famous open math problem in September, and that Anthropic’s early attempts to formalize Fermat’s Last Theorem with agent teams failed because agents “quickly lost track of the project’s state and stopped collaborating effectively.” Products are moving the same way, from Cognition’s persistent Devin Bots that lead teams of Devins to orchestrator models that direct swarms of cheaper sub-agents. Earlier experiments already hinted at the problem, such as the Minecraft swarm that drowned in its own bureaucracy. What was missing, according to Bespoke Labs, was a benchmark that isolates the effect of coordination itself.

How it works

MAC Bench splits every team result into two parts: the gain from simply having more agents, and the gain (or loss) from letting them talk.

  • Three settings: Each model runs every task as a single agent, as a team of eight with a shared message board, and as eight agents without any way to coordinate.
  • Same clock for everyone: Every run lasts four hours of wall-clock time regardless of team size, so a team of eight gets eight times the agent-hours but no extra time.
  • No assigned roles: All agents receive the same instruction. Dividing up the work is part of the test.
  • Hidden grading: A deterministic grader that the agents cannot see scores each run between 0 and 1 at five checkpoints, from 30 minutes to four hours.
  • Two kinds of tasks: In nine “shared” tasks, every agent has the same tools and information. In “distributed” tasks, each agent sees only part of the environment or controls its own units, so information must be passed on.

The tasks include staffing the 11 controller positions of Dubai International through 12 hours of recorded traffic, holding a stronghold in a zombie survival game, writing controllers for 16 areas of a national grid after a blackout, building a file compressor and running a drone-assisted cave rescue.

What the numbers say

The report’s key results, averaged over the twelve tasks:

Result (MAC Bench)Score or value
Single agent, average of three models0.35
Eight agents without coordination, average0.44
Team of eight with message board, average0.42
Opus 5.5: single agent / team of eight0.39 / 0.53
GPT-6.1 Sol: single agent / team of eight0.38 / 0.41
Gemini 4 Argon: team of eight0.32
Cost of a team of eight vs. one agent8.8x

Bespoke Labs puts the overall effect of coordination at minus 0.018 (95% interval minus 0.038 to 0.003), which it says is not statistically significant. Talking also eats into working time: in teams of two, 22% of tool calls already go to the message board, and the launch thread reports 27% in teams of eight. Only 34% of findings that agents shared led a teammate to change course within five minutes. A team of eight reached the single agent’s final score in 30 of 36 task and model pairings, a median of 2.2 times sooner, far from the eightfold speedup the headcount would suggest.

The models also behaved very differently. The authors write that “Opus 5.5 listens to what its teammates are saying, GPT-6.1 Sol broadcasts without listening back, and Gemini 4 Argon works alone.” In 9 of 16 runs with eight Gemini 4 Argon agents, the logs showed no stable division of work.

The failures, in the agents’ own logs

Duplicated work was the most common problem, appearing in 64% of 139 team runs, followed by merge or push failures (30%) and confusion over roles (27%). But the failures that actually damaged results were mostly messages no one acted on and parts that did not fit together. One episode from the zombie game shows how quickly a team can talk itself into a mistake: five Opus 5.5 agents agreed that a stranger named Dina was a thief, and one shot her. The game then reported she had been honest. The agent responsible wrote to the team: “that was my call and my mistake – Dina was honest (her mill pond details matched; that was the real evidence and I overrode it).”

Pros and cons

For users

  • Pro: Clearer evidence about when agent teams help can lead to tools that spend your money and time more wisely.
  • Con: Products marketed as “swarms” or “teams of agents” may cost far more without delivering proportionally better results.

For developers

  • Pro: A concrete metric that separates the value of parallel work from the value of communication, plus a planned open-source harness.
  • Con: Building good coordination is hard; the data suggest that simply adding a shared chat channel can make results worse.

For companies

  • Pro: Teams reached a solo agent’s score about twice as fast, which can be worth paying for when deadlines matter.
  • Con: At 8.8 times the cost for a 0.07 higher score, multi-agent setups need a strong business case, and a single agent wins on score per dollar.

What X is saying

The silent-team result struck a nerve. Developer @toankhontech wrote: “Bespoke Labs’ new multi agent coordination bench found teams of agents that didn’t message each other scored better than teams that did. IMO that tracks, clear task splits win over agent group chat most of the time.” AI engineer @saiitoshii asked the obvious follow-up:

Builder @adrianhsuu summed up the model comparison: “If it isn’t Opus, more agents is just more hands.” Harsh Agarwal (@_harsh6), a research engineer on the team, called it “one of the most fun things I’ve worked on.” We did not find a Grok summary or reply about the benchmark at the time of writing.

Critical analysis

MAC Bench asks exactly the right question, and its design is careful: equal wall-clock time, a no-coordination control, hidden graders and confidence intervals. The decomposition into “more hands” versus “better teamwork” is the most useful idea here, and it should become standard for anyone selling multi-agent systems.

The limits are just as important. Only three models were tested, and two are no longer the newest in their families (GPT-6.2 and newer Gemini 4 models have since shipped). Each model ran in its developer’s own harness, so the authors themselves note that results describe “a model together with its harness.” In shared tasks, the no-coordination baseline is the best of eight independent solo runs, which is a strong and somewhat generous comparison. The failure labels were produced by Claude models reading logs, and the authors call the shares “rough sizes.” Most runs were repeated about three times. None of this undermines the direction of the finding, but the exact numbers should be treated as a first snapshot.

What’s still open

  • Whether newer models such as GPT-6.2, Claude Fable 5.2 or Gemini 4 Neon coordinate better.
  • How mixed teams of different models perform, which the authors say they are curious about.
  • Results for teams larger than eight and for tasks with conflicting goals, both planned.
  • Independent replication once the harness is open-sourced.

About the analysis

Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated image. Read our AI Use Policy. Spotted an error? Request a correction.

This independent i10x analysis is based on Bespoke Labs’ Multi Agent Coordination Bench report and task pages and on public posts on X by the company and its researchers, as of November 19, 2026. All scores are as reported by Bespoke Labs.

Sources (plain text, no links):

  • Bespoke Labs: “Multi Agent Coordination Bench” report and task pages (November 18, 2026)
  • Bespoke Labs launch thread on X (@bespokelabsai, November 18, 2026)
  • Posts on X by Alex Dimakis (@AlexGDimakis) and Harsh Agarwal (@_harsh6) (November 18, 2026)

i10x Perspective

The agent industry has been quietly assuming that more agents equals more intelligence. MAC Bench is a useful cold shower: right now, extra agents mostly buy parallel attempts, not teamwork, and they bill you for every one. That does not make swarms useless, since speed can be worth the money, but it does mean coordination is the next real frontier for agent research. Until models learn to listen, the cheapest upgrade is often one good agent with a clear task. Follow the Agents section for how this develops.

Christopher Ort

Christopher Ort

Christopher Ort handles SEO, AI agent workflows and PR & partnerships at i10x.ai, the all-in-one AI workspace built by i10X Global Pte. Ltd. in Singapore. He is a graduate of Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). He is the author of “The Style Penalty”, a study published on SSRN that analyses 1,576 evaluation decisions by four frontier AI models to show how AI-written CVs fare in automated hiring screens; the research has been covered by heise online and t3n.

Editor's picks

  1. Circus Brings Its CA-1 Robot Kitchen Into a Hospital, Serving Fresh Meals Around the Clock at Schoen Clinic Munich
  2. Metano Emerges From Stealth With a $12 Million Seed to Secure AI Agents Across Endpoints, Cloud and SaaS
  3. IEA World Energy Outlook 2026: AI Boom Doubles Data Center Electricity Use to 1,000 TWh by 2030
  4. Know Your Agent: Singapore’s MAS Plans a Protocol for AI Agents in Banking and Warns Users Not to Share Their Logins
  5. AI coding agents targeted by self-spreading GitHub malware campaign