Reasoning Models Diagnose Better but Add More False Positives, Nature Study Finds

i10X Quick Take

Quick Take

1. The test: Researchers ran six AI models in both “thinking” and “non-thinking” mode on 2,978 real hospital and ICU records, including GPT-5 Mini, Gemini 2.5 Flash, Claude Sonnet 4, DeepSeek-R1/V3 and two Qwen3 models.
2. The trade-off: Reasoning mode found the primary diagnosis more often (52.8% vs. 38.7%), but added more wrong diagnoses (1.47 vs. 1.09 false positives per case) and gave less consistent answers across repeated runs.
3. The fix tested: When 15 physicians checked GPT-5’s reasoning output, their primary accuracy rose by about 20% and the model’s false positives fell by about 44%, in a controlled setting.

Reasoning models are marketed as the better choice for hard problems, and clinical diagnosis looks like a prime example. A peer-reviewed study published on October 29, 2026 in Nature Communications by Shuai Wu, Linwei Chen and colleagues from the University of Hong Kong, Tsinghua University, Harvard Medical School and other institutions shows a catch: the same reasoning that improves accuracy also makes models invent more diagnoses and change their minds more often.

What the study shows

The team built AIMS-LDR, a long-context diagnosis benchmark from real electronic health records. LDR-MIMIC has 1,157 hospital admissions from MIMIC-IV with 40 target diseases, LDR-eICU has 1,821 ICU cases from a multi-center US database with 74 disease categories. Records average 2,657 tokens. Each model received labs, vital signs, history, exam notes and imaging reports and had to name a ranked differential and one to three final diagnoses. Because each provider’s thinking and non-thinking variants come from the same base model, the comparison isolates the effect of reasoning.

• Better primary diagnosis: On LDR-MIMIC, reasoning models reached 52.8% primary accuracy against 38.7% for the non-thinking versions. On LDR-eICU the gap was smaller: 35.1% vs. 28.9%.
• Better differentials: The best reasoning model put the true disease into its top 10 in 84.3% of cases, against 72.6% for the best non-thinking model. Clinically irrelevant predictions fell from 45.0% to 32.8%.
• More false positives: Reasoning models named 1.47 conditions per case that the patient did not have, against 1.09, a 34.8% increase. On LDR-eICU the figures were 5.47 vs. 5.01. As a result, they got the complete set of high-priority diagnoses exactly right less often (9.1% vs. 10.9%).
• A dose-response effect: Turning up GPT-5 Mini’s reasoning effort from minimal to high raised accuracy and false positives together.
• Less reproducible: Across three runs on the same 100 cases, reasoning models kept the same primary diagnosis in 52.3% of cases, non-thinking models in 69.0%. Even at temperature 0, consistency was 55.8% vs. 78.6%.
• Costly errors: 57% of the reasoning models’ false positives and 63% of the non-thinking models’ were classed as potentially harmful. Mapped to real MIMIC-IV care patterns, the reasoning models’ false positives implied more lab tests (106.3 vs. 95.8) and medication classes (57.7 vs. 52.0), and the composite burden index rose from 0.52 to 0.61.

Why it matters: accuracy is not the same as safety

• A real trade-off, not noise: The pattern held across prompt families, temperatures, model sizes, both datasets and an open-ended setting without a candidate list (primary accuracy 37.2% vs. 35.0%, false positives 1.62 vs. 1.42).
• Matched pairs: Comparing thinking and non-thinking modes of the same model families reduces confounding from different training data and architectures.
• Clinical metrics: Beyond accuracy, the authors scored false-positive burden, severity (Elixhauser index) and downstream resource use, which is closer to what matters in a hospital.
• Peer reviewed with code: The paper passed peer review at Nature Communications, and the processing and evaluation code is public on GitHub.

Method in brief

Data: 1,157 MIMIC-IV admissions (1,054 patients) and 1,821 eICU cases; discharge diagnoses, ICD codes and outcome summaries removed; leakage screened with Claude Opus 4.6 and a blinded manual review of 100 cases. Models: GPT-5 Mini, Gemini 2.5 Flash, Claude Sonnet 4, DeepSeek-R1-0528 vs. DeepSeek-V3-0324, Qwen3-235B and Qwen3-30B, each in thinking and non-thinking mode, temperature 1.0, one run per case plus three runs on 100 cases. Physician study: 15 general practitioners (three junior, two mid-career per panel) reviewed GPT-5’s output on 50 respiratory, 50 circulatory and 50 metabolic cases. Statistics: bootstrap 95% confidence intervals.

What remains open

• Retrospective and constrained: The main benchmark uses fixed candidate lists and discharge codes as ground truth, which contain coding noise. The authors call for prospective validation before any clinical use.
• Older models: The paper was submitted in November 2025. GPT-5 Mini, Gemini 2.5 Flash and Claude Sonnet 4 are no longer the newest versions, so absolute numbers may differ for current models.
• Collaboration benefit may be inflated: Physicians got the same 40-disease candidate list. The authors note this could overstate the gain compared with open-ended practice. The GPs were recruited via an annotation company in India.
• The “40%” figure: The abstract speaks of roughly 40% lower reproducibility. The averages in the main text (52.3% vs. 69.0%) give a relative gap of about 24%, and the exact calculation behind the headline figure is not spelled out.
• Review time: Checking full reasoning traces took physicians 15 to 20 minutes per case, against 5 to 8 minutes for short outputs.

Context: reasoning models in medicine

The study adds a clinical dimension to the debate about “thinking” models. Earlier i10X coverage showed how AI diagnoses bend toward patient beliefs. Here, the risk is different: more reasoning makes models more eager to find something. An error analysis of DeepSeek-R1 found that 51.4% of sampled false positives came from reasoning that contradicted the final answer or from secondary findings promoted to primary diagnoses. More on safety research is collected under Safety, Ethics & Society.

What it means for health AI teams

• Measure false positives: Do not choose a reasoning mode on accuracy alone. Track false positives per case and run-to-run consistency.
• Run cases more than once: A single answer from a reasoning model can change on the next run. Repeated runs or consensus methods reveal unstable cases.
• Keep the physician as the filter: The tested setup, AI first and clinician as guardrail, improved both accuracy and safety. Autonomous use is not supported by these results.
• Budget review time: Long reasoning traces help verification but cost time. Interfaces should condense them without hiding the evidence.

Model details are on the OpenAI, Gemini and Claude hubs.

About the analysis

Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.

This article is based on the full text of “The accuracy-safety trade-off of large reasoning models in clinical diagnostics” (Nature Communications, article in press, published October 29, 2026, doi:10.1038/s41467-026-78520-x), including Methods and the limitations section. All numbers come from the paper; i10X did not rerun the experiments. The paper is licensed CC BY-NC-ND 4.0, so the chart was drawn by i10X from the reported values rather than reusing the original figures. AI tools assisted with research and drafting; the selection, fact-checking and final editing were done by the i10X editorial team.

i10x Perspective

For model providers, “think longer” is usually sold as pure upside. This study is a useful counterweight: in a domain where a wrong extra diagnosis triggers tests and drugs, more reasoning without calibration can be a liability. The strongest result is not that reasoning models fail, but that they work best with a human who knows when to say no.