Benchmarks & EvalsGoogleGemini
Evidence check
  • Preprint
  • No code released
  • Not independently replicated

Study: AI Agents Reading CT and MRI Scans Fall Far Short of Radiologists

Six AI models scrolled through 100 emergency CT and MRI scans like radiologists. They were right about a third of the time, latched onto early guesses and stayed about 90% confident.

Christopher Ort
By Christopher Ort · 7 min read
Share
A radiologist reviews grayscale CT scan images on diagnostic monitors in a dim hospital reading room
Image: AI-generated, i10X
On this page
  1. Quick Take
  2. In plain English
  3. What the study shows
  4. Why it matters: a realistic reading test with human benchmarks
  5. Method in brief
  6. What remains open
  7. Context: a growing body of sobering medical AI results
  8. What it means for hospitals and patients
  9. About the analysis
  10. i10x Perspective
i10X Quick Take

Quick Take

  1. The finding: When six vision-language models had to scroll through 100 emergency CT and MRI exams on their own, they named the correct diagnosis in 32.6% of runs. On 30 cases that nine radiologists also read, the freely navigating models scored 30.0% versus 77.0% for the radiologists.
  2. Why it matters: The models usually formed a first guess after seeing about a tenth of the slices and kept it 63.1% of the time. When that first guess was wrong, only 12.4% of runs still ended with the right answer.
  3. What it means: Median stated confidence was 90%, while only 38.9% of answers were correct. The authors warn that strong scores on curated image sets do not translate into reliable autonomous reading of real scans.

In plain English

A CT or MRI scan is not one picture but a stack of dozens or hundreds of slices that a radiologist scrolls through. Researchers let six AI models do that job on 100 real emergency cases. The models got the diagnosis right only about one time in three, far below human radiologists, who were right in about three out of four reads. The models also tended to stick with their first guess and sounded very sure of themselves even when they were wrong. For patients, the message is simple: an AI that sounds confident about a scan is not necessarily right.

What the study shows

Most tests of medical AI show a model a few hand-picked key images and ask for a diagnosis. Real radiology works differently: the reader decides where to look, scrolls through series and revises ideas along the way. A team led by Jonathan Lichtle and Aymen Meddeb, with senior author Bastien Le Guellec, from Lille University Hospital, Charité Berlin, Cheikh Anta Diop University Hospital in Dakar and the Technical University of Munich hospital, built an environment in which AI agents read scans exactly that way. Their preprint was posted to arXiv on November 17, 2026.

  • Navigating on their own hurts: Pooled across six models, accuracy was 32.6% under free navigation, 37.9% when the agent started on the most informative slice, and 42.2% or 42.5% when all slices or only the key slices were sent in a single request. Free navigation was the least accurate condition for four of the six models and tied for a fifth.
  • Far behind radiologists: On the 30 shared cases, models were 31.8 to 47.0 percentage points less accurate than radiologists in every condition. Graded by two radiologists instead of the AI judge, the radiologists reached 82.6%.
  • They looked at the right place: 95.7% of free-navigation runs displayed a key slice before answering, including 95.3% of runs that ended wrong. Unlike radiologists, the models did not linger on those slices. The authors conclude that interpreting the images, not finding them, was the bottleneck.
  • Early guesses stick: The first working hypothesis appeared after a median of 10.8% of slices and was correct in 26.0% of runs. Final answers were right in 90.4% of runs after a correct first guess but in only 12.4% after a wrong one.
  • Much of the answer came from text: With no images at all, only slice labels, models were still correct in 17.2% of runs, and 11.3% with the clinical indication alone. Only 5.5% of the image-free answers mentioned that images were missing.
  • Confidence did not track accuracy: Correct and incorrect answers came with almost the same median confidence (90% versus 88%). Calibration was worse than always stating the average accuracy, in every model and every condition.
Grouped bar chart of diagnostic accuracy on 100 emergency CT and MRI exams with all slices in one request versus free navigation: pooled 42.2% vs 32.6%, Gemini 3.7 Flash 52.5% vs 44.0%, GLM-5.3 Flash 52.5% vs 38.5%, GPT-5.6 Luna 41.5% vs 30.5%, Qwen 3.7 Flash 34.5% vs 34.5%, MiMo v2.5 33.5% vs 26.0%, MiniMax M3 38.5% vs 22.0%.
i10X chart based on data from Lichtle, Meddeb et al. (2026), arXiv:2611.23480, Table 1 and Table S4. Own chart, no original figure reused (paper licensed CC BY-NC-SA 4.0).
Reading conditionCorrect runsAccuracy (95% CI)
Free navigation391 of 1,20032.6% (26.7 to 38.8)
Navigation from the key slice455 of 1,20037.9% (31.8 to 44.4)
All slices, single request506 of 1,20042.2% (35.7 to 48.8)
Key slices, single request510 of 1,20042.5% (36.2 to 48.9)
Slice labels, no images103 of 60017.2% (12.0 to 22.5)
Radiologists (30 cases)208 of 27077.0% (65.9 to 87.0)
Selected rows from Table 1 of Lichtle, Meddeb et al. (2026), arXiv:2611.23480. Model rows cover all 100 cases; the radiologist row covers the 30 cases they read.

Why it matters: a realistic reading test with human benchmarks

The agents used the same viewer as the radiologists: one slice at a time, the same series switches, the same windows and the same one-line clinical indication. Every action was logged, so the study could compare not only answers but behavior. Cases came from a public teaching archive published between March 2025 and August 2026, and the authors checked for training-data leakage by comparing cases published before and after each model’s stated knowledge cutoff. Accuracy did not differ, and on the 34 cases published after the latest stated cutoff, free navigation was still the weakest condition.

Grading was validated carefully. The AI judge, Claude Sonnet 5, came from a developer whose models were not tested, and on 500 runs it agreed with two radiologists’ consensus as closely as each radiologist did (kappa 0.83 versus 0.84 and 0.83). The statistics account for the fact that runs on the same case are correlated, report bootstrapped confidence intervals and follow the CLAIM checklist for medical imaging AI. The agent code is public.

Method in brief

  • Cases: 100 emergency CT and MRI exams (73 CT, 26 MRI, 1 combined): 31 abdominal, 30 neurologic, 20 thoracic, 19 musculoskeletal and trauma, including 7 normal studies.
  • Models: Gemini 3.7 Flash (Google), GPT-5.6 Luna (OpenAI), GLM-5.3 Flash (Z.ai), Qwen 3.7 Flash (Alibaba), MiniMax M3 and MiMo v2.5 (Xiaomi), chosen by LMArena rank among models under US$2 per million output tokens, run through OpenRouter at temperature 0.
  • Runs: Four reading conditions, two runs each: 4,800 runs in total, plus 1,200 no-image control runs.
  • Agent loop: At each turn the agent described the slice, updated its understanding, planned its next step, stated a working hypothesis with a confidence score and moved one slice, switched series or answered. Only the three most recent images stayed in memory; keeping all images did not change accuracy.
  • Human comparison: Nine radiologists from four institutions (three senior residents, six board-certified, median 7 years of experience) read 30 of the cases in the same viewer.

What remains open

  • Teaching cases, not clinical practice: The exams came from an online teaching archive, and some were published before the models’ knowledge cutoffs, although accuracy did not differ before and after.
  • Not the biggest models: The thousands of runs required kept the authors to cheaper models, so the largest versions were not tested. Images were also downscaled to at most 512 pixels on the longest side.
  • The setup may encourage early commitment: Agents had to state a hypothesis at every turn, which stayed in their context and may itself have nudged them to commit early, the authors note.
  • Only the top diagnosis counted: Real reports usually list differential diagnoses; the study graded only the primary one.
  • Radiologists disagreed too: Agreement among the nine radiologists was only fair (Fleiss kappa 0.22). The paper is a preprint, received no specific funding and contains no competing-interests statement.

Context: a growing body of sobering medical AI results

The findings echo text-based studies in which language models diagnosed less accurately when they had to gather patient information themselves, and the MIRAGE study, in which frontier models described images they had never received. Two recent preprints with agents in real viewers point the same way: in ABRA, agents ran viewer commands almost perfectly but often failed to localize findings, and MedOpenClaw showed that answer-only scores overstate competence once the supporting evidence is checked. On i10X we recently covered a Nature study on reasoning models in clinical diagnosis, which found better accuracy but more false positives. This paper adds a behavioral picture: the models search competently, then anchor.

What it means for hospitals and patients

  • Do not read confidence as reliability: A model saying it is 90% sure meant little here. Any clinical tool built on such models needs independent calibration checks.
  • Test agents, not just models: Accuracy changed just by changing the starting slice. Hospitals evaluating AI reading tools should test them in realistic, interactive workflows, not on curated image sets.
  • Watch for text shortcuts: A large part of the models’ answers came from the indication and exam type rather than the pixels. Evaluations should include no-image controls like this one.
  • Patients using chatbots: Uploading scan images to a general chatbot is not a substitute for a radiologist’s report, based on this evidence.

Find more evaluations like this on our Benchmarks & Evals studies page and in our Health & Medical Research section.

About the analysis

Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated image. Read our AI Use Policy. Spotted an error? Request a correction.

This article is based on the full text of Lichtle, Meddeb, Birot, Niang, Gastaud, Hanafi, Dujardin, Selomme, Cachard, Hamroun, Kuchcinski, Kim and Le Guellec (2026), “Diagnostic Accuracy and Navigation Behavior of Vision-Language Model Agents Reading Emergency CT and MRI Examinations,” arXiv:2611.23480v1 (cs.AI, submitted November 17, 2026), including the supplemental material. All numbers come from the paper. The chart was drawn by i10X from Table 1 and Table S4; the paper is licensed CC BY-NC-SA 4.0, so no original figures were reused. Agent and analysis code: github.com/bastienLeGuellec/radioagent. The study received no specific funding. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.

i10x Perspective

This is one of the more useful medical AI papers in a while because it tests the job, not a quiz. The striking result is not the low accuracy of cheaper models, but how they fail: they look at the right slice, decide too early and then sound certain. That is a failure mode a benchmark score will not show you. Before AI agents read scans unsupervised, developers need to prove they can change their mind.

Christopher Ort

Christopher Ort

Christopher Ort handles SEO, AI agent workflows and PR & partnerships at i10x.ai, the all-in-one AI workspace built by i10X Global Pte. Ltd. in Singapore. He is a graduate of Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). He is the author of “The Style Penalty”, a study published on SSRN that analyses 1,576 evaluation decisions by four frontier AI models to show how AI-written CVs fare in automated hiring screens; the research has been covered by heise online and t3n.

Timeline

What happened before

  1. Release

    Gemini 4 Argon

    1.05M context · $2 / $10 per 1M tokens

    Google's most capable model for complex coding, agentic, math and science tasks.

    ai.google.dev (source, opens in a new tab)
  2. Release

    Gemini 3.8 Flash

    1.05M context · $0.75 / $3.75 per 1M tokens

    ai.google.dev (source, opens in a new tab)
  3. Release

    Gemini 3.7 Flash

    1.05M context · $0.75 / $3.75 per 1M tokens · Deprecated 8 Oct 2026 (API routes to Gemini 3.8 Flash)

    ai.google.dev (source, opens in a new tab)
  4. Release

    Gemini 3.6 Flash & 3.5 Flash-Lite

    1.05M context · $0.75 / $3.75 per 1M tokens

    ai.google.dev (source, opens in a new tab)
  5. Release

    Gemini 3.5 Flash

    1.05M context · $1.5 / $9 per 1M tokens · Deprecated 8 Oct 2026 (API routes to Gemini 3.6 Flash)

    Launched at I/O 2026 as the first Gemini 3.5 model; Google later dropped the planned 3.5 Pro.

    blog.google (source, opens in a new tab)
Full Gemini timeline ›

Editor's picks

  1. AI coding agents targeted by self-spreading GitHub malware campaign
  2. GHSA-xgh8-qwjh-q853: vLLM chat template bypass in /v1/systemone endpoint
  3. OpenAI’s Dean Ball Rejects the ‘Silent Consensus’ That AI Will End Human Primacy
  4. Alibaba Earnings: AI Cloud Revenue Jumps 53% as External Customers Grow 64% and Capex Rises 60%
  5. Canva AI Controls: Users Can Now Turn Off Generative AI Features and See Less AI-Generated Content