Every office has that colleague who replies to every Slack thread, needed or not. It turns out most AI models are that colleague. Deepnote’s new Agent Initiative Benchmark, released on November 13, 2026, dropped 12 frontier models into the Slack of a synthetic software company and measured whether they knew when to help, how to help, and when to stay quiet. OpenAI models took the top two spots, with GPT-6 Astra at 8.97 out of 10 and GPT-6.1 Sol at 8.65, ahead of Anthropic‘s Claude Fable 5.2 at 8.17. The more useful finding for anyone building agents: models are good at helping and bad at not helping.
Quick Take
- What it is: An open benchmark from Deepnote that tests “proactive” agents: AI that watches a company’s Slack and steps in only when useful, across 50 situations and eight task types.
- The test bed: PageMantle, a synthetic company with six years of connected records and about 10 GB of data across 11 services, including Slack, Notion, Linear, HubSpot, Stripe, PostHog, BigQuery and Snowflake.
- The ranking: GPT-6 Astra (8.97), GPT-6.1 Sol (8.65), Claude Fable 5.2 (8.17), Claude Opus 5.5 (7.84). Grok 4.7 finished last at 4.51, Gemini 4 Argon second to last at 5.60.
- Cost shock: Sol cost $0.11 per task against $1.87 for Fable 5.2. GPT-6 Luna and Claude Fable 5.1 are 430 times apart in price but only 1.1 points apart in score.
- Biggest weakness: When silence was the right answer, models held back only 6 times out of 10 on average.
In plain English
Companies want AI helpers that sit in their chat tools and jump in when something goes wrong, like a sales team promising a launch date that engineering just moved. Deepnote built a fake company and tested 12 AI models on 50 such moments. OpenAI’s models did best, and some cheap models did nearly as well as very expensive ones. The most common mistake was not staying silent when nobody needed help.
What happened
Deepnote, which builds a data workspace “where agents and people work on data together,” published the benchmark with an open dataset on Hugging Face and simulated service APIs plus example tasks on GitHub. Founder Jakub Jurovych introduced it on X with the question “What’s the best model for proactive agents?”
The setup is deliberately identical for every model: same instructions, same tools, same company snapshot, high reasoning effort and a 180-second budget of model response time (tool execution is not counted). In each run the agent sees the newest Slack messages and one standing instruction: assess what is going on and help if it is useful. It can search and read across all 11 services, including writing SQL against BigQuery and Snowflake, then post in a channel, reply in a thread, send a direct message, or do nothing. Two LLM evaluators, blind to which model they are grading, score each run from 0 to 10 against a reference solution on four dimensions: intervention or silence, content and evidence, clarity and tone, and audience and location.
The results
| Model | Score (0-10) | Cost per task | Avg. time | Tool calls | Stayed silent |
|---|---|---|---|---|---|
| GPT-6 Astra | 8.97 | $0.693 | 54 s | 19.3 | 8/10 |
| GPT-6.1 Sol | 8.65 | $0.112 | 54 s | 18.0 | 8/10 |
| Claude Fable 5.2 | 8.17 | $1.87 | 60 s | 20.9 | 6/10 |
| Claude Opus 5.5 | 7.84 | $1.28 | 29 s | 10.0 | 8/10 |
| Claude Fable 5.1 | 7.50 | $3.02 | 51 s | 14.5 | 6/10 |
| Claude Sonnet 5.5 | 6.65 | $0.420 | 15 s | 8.9 | 6/10 |
| GPT-6 Luna | 6.38 | $0.0070 | 50 s | 13.7 | 4/10 |
| DeepSeek V4.1 Flash | 6.35 | $0.026 | 86 s | 21.1 | 4/10 |
| Mistral Large 4 | 6.09 | $0.082 | 85 s | 22.5 | 4/10 |
| Claude Haiku 5.5 | 6.05 | $0.075 | 28 s | 12.1 | 5/10 |
| Gemini 4 Argon | 5.60 | $0.403 | 164 s | 34.5 | 10/10 |
| Grok 4.7 | 4.51 | $0.425 | 145 s | 35.5 | 6/10 |
Source: Deepnote Research. “Stayed silent” counts the 10 tasks where the right move was to say nothing. Cost is recorded model spend per task.
Deepnote is careful about uncertainty. Sol’s score sits inside Astra’s 95% interval, “so this task set does not separate them,” the company writes. Fable 5.2 is below Astra’s interval but inside Sol’s. The bottom of the table is driven by timeouts: Gemini 4 Argon and Grok 4.7 hit the 180-second limit without sending a message in 27 and 18 runs, and scored zero on 17 and 20 tasks.
What the benchmark reveals
- Expensive is not better. “The three priciest models finish third, fourth, and fifth,” Deepnote writes. A month of watching about 2,500 Slack threads would cost $281 on Sol and $4,684 on Fable 5.2, by its estimate.
- Silence is the hard part. When help was needed, a message went out 9 times in 10. When silence was right, models held back only 6 times in 10. Speaking out of turn also cost about 1.8 times as much as staying quiet.
- Two working styles. Deepnote calls them “snipers or deep thinkers.” Opus 5.5 answers in 29 seconds with 10 tool calls. Fable 5.2 makes twice the calls in twice the time for a score 4% higher.
- More tool calls, lower scores. Gemini 4 Argon averaged 35 tool calls, Grok 4.7 averaged 36, and both landed at the bottom.
- Numbers are the weak spot. Analyze tasks, which require pulling figures from analytics or warehouse data and checking them against a claim in Slack, were the weakest type for 8 of 12 models.
One example task shows the difference. A customer success manager asks which of three customers needs attention first. Only models that opened a report linked from a Linear ticket found that one customer’s preview pages were wrong. GPT-6.1 Sol and Claude Fable 5.2 scored 10; Gemini 4 Argon and Grok 4.7 hit the time limit after 48 and 56 tool calls without replying.
Pros and cons
For users (teams working with agents)
- Pro: Clear evidence that a well-chosen model can catch real coordination mistakes, like a sales promise that contradicts the release plan.
- Con: Even the best models spoke up in 2 of 10 situations where they should have stayed quiet. Deepnote notes those messages were well written, so they are hard to filter out.
For developers
- Pro: Open data and simulated APIs to test your own agent, plus practical advice: route by task complexity, make “is this already handled?” an explicit step, and cap the search budget.
- Con: Results compare models under one fixed harness. Deepnote itself warns that “a different prompt, tool policy, or retrieval step will move every number here.”
For companies
- Pro: A cost lens that matters for always-on agents, where a model runs on every thread, not once per question.
- Con: 50 tasks, one effort level and one synthetic company. Rankings by task type rest on 5 to 8 tasks each and should be treated as tendencies.
What X is saying
Jakub Jurovych’s launch thread summed up the headlines:
What’s the best model for proactive agents?
— Jakub Jurovych (@jakubjurovych) November 13, 2026 · View on X
Today, we’re introducing the Agent Initiative Benchmark. We tested 12 frontier models on realistic workplace tasks inside a synthetic company with 6 years of highly detailed history: it has a financial model, CRM records, past-due invoices, Linear tickets, and real-life events affecting company development – competitor product announcements, SVB collapse, etc.
We used the company’s Slack to test how a proactive agent listening in on each conversation would assist the team in day-to-day tasks: from stepping in to align initiatives across teams to warning the team not to use an unapproved customer logo.
Highlights:
– 6.1 Sol and Astra from @OpenAIDevs beat Anthropic’s latest Fable 5.2, with Sol crushing it at both performance and cost ($0.11 per task vs $1.87).
– The most expensive models aren’t the best: Fable 5.1 is 430x more expensive than GPT-6 Luna, with similar performance.
– Most models try to help even if they’re not needed. Interestingly, in these failure cases, they spend more time thinking, and the unnecessary work takes about 80% more time and money than if they stayed quiet.
If you’re building agents, keep reading for more tips on which model is best for your use case. Detailed breakdown, the open-source dataset, and benchmark code below:
The price gap got the most attention:
GPT-6 Luna is 430x cheaper than Claude Fable 5.1 as an always-on Slack agent , and only 1.1 points behind
— Chetaslua (@chetaslua) November 13, 2026 · View on X
> Luna : 6.38/10 , $0.007 a task
> Fable 5.1 : 7.50/10 , $3.02 a task
a proactive agent that reads every thread can’t cost $3 a read
Santiago (@svpino), a longtime Deepnote user by his own account, broke down the test design:
This is a really good test for agents:
— Santiago (@svpino) November 13, 2026 · View on X
It’s a benchmark released by Deepnote. (I’ve used their platform for 5+ years and I love it!)
The benchmark tests agents in a simulated company’s Slack:
• Agents get access to records across 11 services
• They face 50 different situations
• They have 3 minutes to investigate and decide
Here is what they test:
1. Does the agent know when to help?
2. Does the agent know how to help accurately?
3. Does the agent know when to stay quiet?
Others turned it into office humor. “Every team already has that one coworker. Now we pay per token for it,” @rfanazhari wrote. @adrianhsuu noted the week’s irony: “Fable 5.2 won the week. It’s still not the model I’d leave in Slack.” i10x did not find a Grok assessment of the benchmark on X.
Critical analysis
This is one of the more thoughtful agent benchmarks released this year. It publishes intervals, zero counts, costs and limitations, and it tests a behavior most evaluations ignore: restraint. Still, three caveats apply. First, scoring relies on LLM judges, which can share blind spots with the models they grade, even when blinded. Second, Deepnote builds an agent workspace and is not a neutral party in the agent market, though the benchmark compares models rather than products. Third, the 180-second limit shapes the ranking. Models that investigate thoroughly but slowly, such as Gemini 4 Argon, are punished hard, which is realistic for a live Slack thread but less so for background work. Deepnote acknowledges this and says a harness benchmark is next.
For Gemini and Grok, the result is a warning sign rather than a verdict: both lost most of their points to timeouts, not to bad answers.
What’s still open
- How rankings change at lower or higher reasoning effort. Deepnote ran only “high.”
- Whether a router that sends routine threads to cheap models and hard ones to strong models beats any single model. Deepnote recommends it but did not test it.
- How newer models, including the rumored GPT-6.2 release, perform on the same tasks.
- Whether results hold with real company data rather than a synthetic company.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated image. Read our AI Use Policy. Spotted an error? Request a correction.
This independent i10x analysis is based on Deepnote’s benchmark page, its GitHub repository and public posts on X. i10x did not rerun the benchmark.
i10x Perspective
The leaderboard will change with the next model release. The finding about silence will not. An always-on agent that is right 9 times in 10 but interrupts when it should not will be muted by its team within a week, however polished its messages. Deepnote’s best advice is also its simplest: make “is this already handled?” an explicit step, and track false alarms as their own metric. For teams choosing a model today, the cost table is the real headline. Sol delivers near-top quality at a fraction of the price, which makes always-on agents affordable for companies that could not justify $4,700 a month for one Slack bot.
