Quick Take
- The finding: When MCP tools fail the way real MCP servers do, tool-calling agents lose more than half their quality: the task-completion score fell 60%, from 8.08 to 3.23 out of 10.
- Scale does not help: OpenAI’s gpt-oss-120B kept no larger share of its fault-free score than gpt-oss-20B (p = 0.96) and completed descriptively fewer tasks under faults, 91.1% vs. 95.6%.
- The framework matters: LangChain absorbed faults somewhat better than the OpenAI Agents SDK and the MCP-Bench loop, which the authors call suggestive evidence that error handling beats model size.
Agent benchmarks usually assume that every tool answers correctly and on time. Production systems do not work that way: servers return errors, hang, or send malformed data. A paper accepted at NeurIPS 2026 by researchers from the University of Luxembourg (SnT), Tilburg University, the University of Naples Federico II and the Gran Sasso Science Institute measures what realistic tool failures do to agents built on the Model Context Protocol (MCP). The short answer: a lot, and a bigger model is no protection.
What the study shows
The team built MCP-Injector, a middleware that sits between an agent and its MCP servers and injects faults without changing either side. Which faults to inject came from the ecosystem itself: 57,240 GitHub issue and pull-request threads from 1,774 open-source MCP server repositories, classified by failure type and persistence. Explicit errors that persist until someone fixes them dominate (65.6% of the injection profile), followed by silent wrong or malformed content (19.1%) and timing failures such as delays and hangs (9.7%).
| Judge metric (0 to 10) | No faults | Calibrated faults | Drop |
|---|---|---|---|
| Task completion | 8.08 | 3.23 | −60% |
| Tool selection | 8.10 | 3.36 | −59% |
| Planning effectiveness | 7.51 | 2.49 | −67% |
| Grounding | 8.09 | 3.10 | −62% |
| Parameter accuracy | 8.00 | 2.91 | −64% |
| Parallelism efficiency | 7.08 | 2.23 | −68% |
- Silent damage, not crashes: Most runs still finished. Completion rates fell from 95 to 100% without faults to 88 to 97% with faults. The typical outcome (56% of runs) was partial degradation: an answer, but a noticeably worse one. Among runs that completed in both conditions, the task-completion score still fell 57%.
- Bigger model, more timeouts: The 120B model finished fewer tasks under faults in every stack, most clearly with the OpenAI Agents SDK (88% vs. 95%). Its fault-attributable drop was 5.8 points vs. 3.6 for the 20B, but the difference between the two is not statistically clear. When it did finish, its answers scored higher (3.68 vs. 3.26).
- Answers without tools: In 6.0% of faulted runs, agents produced a plausible answer while barely using tools. One run computed every unit conversion from memory and scored 9.3 for task completion. Counting only grounded answers, the fulfillment drop rises from 58% to 63%.
- Task type matters: Tasks relying on live APIs (−97%) and long sequential tool chains (−82%) suffered most; short computational tasks least (−19%).
Why it matters: a missing dimension in agent evaluation
- Faults from the field: Instead of invented failure scenarios, the fault mix follows reported incidents. Two senior researchers checked the automatic classification on 117 threads with high agreement (Cohen’s kappa 0.93 for failure type, 0.80 for persistence).
- Clean comparison: Every faulted run is paired with the same task and seed run through the proxy without faults. The proxy itself did not change results when idle (no significant difference in 18 corrected tests).
- Careful wording: The authors label the framework effect as suggestive and the completion gap between models as descriptive, and name the alternative explanations themselves.
- Open artifacts: Code, data and analysis pipeline are on GitLab and archived on Zenodo.
Method in brief
Benchmark: 25 single-server tasks generated with the MCP-Bench pipeline from 25 MCP servers, given as naturalistic, underspecified prompts. Models: gpt-oss-20B and gpt-oss-120B at MXFP4 precision with a 16,000-token context, served locally on Apple Silicon via LM Studio. Stacks: MCP-Bench’s plan-and-execute loop, the OpenAI Agents SDK and a LangChain/LangGraph agent. Conditions: no proxy, proxy without faults, and proxy with the calibrated 18-rule fault profile; three seeds each, 1,350 runs in total. Scoring: gpt-4o-mini as judge, five scorings per run with shuffled rubric order, median reported; 3,000-second timeout per task.
What remains open
- One model family: Both backbones are gpt-oss under one serving setup. Whether other model families or frontier API models are more resilient is untested.
- A stress test, not a failure rate: GitHub issues over-report persistent, reproducible bugs, so the profile is 91% permanent faults. Transient and intermittent faults fired only 9 times out of 1,905 events, too few to study.
- Why the bigger model struggled: The 120B received more faults per run (4.6 vs. 3.9) and wrote longer outputs within the same 16K context. Both are candidate explanations; neither is established.
- Easy tasks, one server: Without faults the benchmark is easy, and multi-server, long-horizon and human-in-the-loop setups are not covered. The judge is itself an LLM and rewards some ungrounded answers.
Context: capability benchmarks hide operational brittleness
Most tool-use benchmarks, from MCP-Bench to earlier function-calling suites, measure success when every tool works. Security work on MCP focuses on attackers; this paper deals with everyday failures such as misconfiguration, outages and schema drift. The closest prior work, ReliabilityBench, also injects timeouts and rate limits, but without a fault mix calibrated to the MCP ecosystem or persistence modes. For more on how agents are built and tested, see i10X agent research.
What it means for teams building agents
- Test with broken tools: Add fault injection to pre-deployment tests. A good benchmark score says little about what happens when a server returns errors for an hour.
- Invest in the harness: Retry policies, switching to alternative tools and clear error surfacing appear to matter more than a larger model. The stacks differed visibly: the MCP-Bench loop re-plans with another tool, while the ReAct-style stacks more often stop calling tools.
- Check grounding: Flag answers produced without successful tool calls; they can look good to a judge and still be made up.
For the models and SDK tested here, see the OpenAI hub.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of “Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance” (arXiv:2610.32555v1, submitted October 23, 2026; NeurIPS 2026), including the limitations and appendix tables. All numbers come from the paper; i10X did not rerun the experiments. The authors thank SURF for computing resources and declare no competing interests. The paper uses the arXiv non-exclusive license, so no original figures are reproduced; both charts were redrawn by i10X from Tables 4 and 5. AI tools assisted with research, drafting and the title illustration; the selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
MCP made connecting agents to tools easy. This paper is a reminder that it also connects them to every bug and outage in those tools. The most useful result is not the 60% drop, which depends on a deliberately harsh fault mix, but the pattern behind it: agents rarely crash, they quietly get worse, and some start answering from memory. That is the failure mode users notice last. If the framework finding holds up beyond gpt-oss, the cheapest reliability gain for agent builders is better error handling, not a bigger model.
