Quick Take
- The finding: When six vision-language models had to scroll through 100 emergency CT and MRI exams on their own, they named the correct diagnosis in 32.6% of runs. On 30 cases that nine radiologists also read, the freely navigating models scored 30.0% versus 77.0% for the radiologists.
- Why it matters: The models usually formed a first guess after seeing about a tenth of the slices and kept it 63.1% of the time. When that first guess was wrong, only 12.4% of runs still ended with the right answer.
- What it means: Median stated confidence was 90%, while only 38.9% of answers were correct. The authors warn that strong scores on curated image sets do not translate into reliable autonomous reading of real scans.
In plain English
A CT or MRI scan is not one picture but a stack of dozens or hundreds of slices that a radiologist scrolls through. Researchers let six AI models do that job on 100 real emergency cases. The models got the diagnosis right only about one time in three, far below human radiologists, who were right in about three out of four reads. The models also tended to stick with their first guess and sounded very sure of themselves even when they were wrong. For patients, the message is simple: an AI that sounds confident about a scan is not necessarily right.
What the study shows
Most tests of medical AI show a model a few hand-picked key images and ask for a diagnosis. Real radiology works differently: the reader decides where to look, scrolls through series and revises ideas along the way. A team led by Jonathan Lichtle and Aymen Meddeb, with senior author Bastien Le Guellec, from Lille University Hospital, Charité Berlin, Cheikh Anta Diop University Hospital in Dakar and the Technical University of Munich hospital, built an environment in which AI agents read scans exactly that way. Their preprint was posted to arXiv on November 17, 2026.
- Navigating on their own hurts: Pooled across six models, accuracy was 32.6% under free navigation, 37.9% when the agent started on the most informative slice, and 42.2% or 42.5% when all slices or only the key slices were sent in a single request. Free navigation was the least accurate condition for four of the six models and tied for a fifth.
- Far behind radiologists: On the 30 shared cases, models were 31.8 to 47.0 percentage points less accurate than radiologists in every condition. Graded by two radiologists instead of the AI judge, the radiologists reached 82.6%.
- They looked at the right place: 95.7% of free-navigation runs displayed a key slice before answering, including 95.3% of runs that ended wrong. Unlike radiologists, the models did not linger on those slices. The authors conclude that interpreting the images, not finding them, was the bottleneck.
- Early guesses stick: The first working hypothesis appeared after a median of 10.8% of slices and was correct in 26.0% of runs. Final answers were right in 90.4% of runs after a correct first guess but in only 12.4% after a wrong one.
- Much of the answer came from text: With no images at all, only slice labels, models were still correct in 17.2% of runs, and 11.3% with the clinical indication alone. Only 5.5% of the image-free answers mentioned that images were missing.
- Confidence did not track accuracy: Correct and incorrect answers came with almost the same median confidence (90% versus 88%). Calibration was worse than always stating the average accuracy, in every model and every condition.

| Reading condition | Correct runs | Accuracy (95% CI) |
|---|---|---|
| Free navigation | 391 of 1,200 | 32.6% (26.7 to 38.8) |
| Navigation from the key slice | 455 of 1,200 | 37.9% (31.8 to 44.4) |
| All slices, single request | 506 of 1,200 | 42.2% (35.7 to 48.8) |
| Key slices, single request | 510 of 1,200 | 42.5% (36.2 to 48.9) |
| Slice labels, no images | 103 of 600 | 17.2% (12.0 to 22.5) |
| Radiologists (30 cases) | 208 of 270 | 77.0% (65.9 to 87.0) |
Why it matters: a realistic reading test with human benchmarks
The agents used the same viewer as the radiologists: one slice at a time, the same series switches, the same windows and the same one-line clinical indication. Every action was logged, so the study could compare not only answers but behavior. Cases came from a public teaching archive published between March 2025 and August 2026, and the authors checked for training-data leakage by comparing cases published before and after each model’s stated knowledge cutoff. Accuracy did not differ, and on the 34 cases published after the latest stated cutoff, free navigation was still the weakest condition.
Grading was validated carefully. The AI judge, Claude Sonnet 5, came from a developer whose models were not tested, and on 500 runs it agreed with two radiologists’ consensus as closely as each radiologist did (kappa 0.83 versus 0.84 and 0.83). The statistics account for the fact that runs on the same case are correlated, report bootstrapped confidence intervals and follow the CLAIM checklist for medical imaging AI. The agent code is public.
Method in brief
- Cases: 100 emergency CT and MRI exams (73 CT, 26 MRI, 1 combined): 31 abdominal, 30 neurologic, 20 thoracic, 19 musculoskeletal and trauma, including 7 normal studies.
- Models: Gemini 3.7 Flash (Google), GPT-5.6 Luna (OpenAI), GLM-5.3 Flash (Z.ai), Qwen 3.7 Flash (Alibaba), MiniMax M3 and MiMo v2.5 (Xiaomi), chosen by LMArena rank among models under US$2 per million output tokens, run through OpenRouter at temperature 0.
- Runs: Four reading conditions, two runs each: 4,800 runs in total, plus 1,200 no-image control runs.
- Agent loop: At each turn the agent described the slice, updated its understanding, planned its next step, stated a working hypothesis with a confidence score and moved one slice, switched series or answered. Only the three most recent images stayed in memory; keeping all images did not change accuracy.
- Human comparison: Nine radiologists from four institutions (three senior residents, six board-certified, median 7 years of experience) read 30 of the cases in the same viewer.
What remains open
- Teaching cases, not clinical practice: The exams came from an online teaching archive, and some were published before the models’ knowledge cutoffs, although accuracy did not differ before and after.
- Not the biggest models: The thousands of runs required kept the authors to cheaper models, so the largest versions were not tested. Images were also downscaled to at most 512 pixels on the longest side.
- The setup may encourage early commitment: Agents had to state a hypothesis at every turn, which stayed in their context and may itself have nudged them to commit early, the authors note.
- Only the top diagnosis counted: Real reports usually list differential diagnoses; the study graded only the primary one.
- Radiologists disagreed too: Agreement among the nine radiologists was only fair (Fleiss kappa 0.22). The paper is a preprint, received no specific funding and contains no competing-interests statement.
Context: a growing body of sobering medical AI results
The findings echo text-based studies in which language models diagnosed less accurately when they had to gather patient information themselves, and the MIRAGE study, in which frontier models described images they had never received. Two recent preprints with agents in real viewers point the same way: in ABRA, agents ran viewer commands almost perfectly but often failed to localize findings, and MedOpenClaw showed that answer-only scores overstate competence once the supporting evidence is checked. On i10X we recently covered a Nature study on reasoning models in clinical diagnosis, which found better accuracy but more false positives. This paper adds a behavioral picture: the models search competently, then anchor.
What it means for hospitals and patients
- Do not read confidence as reliability: A model saying it is 90% sure meant little here. Any clinical tool built on such models needs independent calibration checks.
- Test agents, not just models: Accuracy changed just by changing the starting slice. Hospitals evaluating AI reading tools should test them in realistic, interactive workflows, not on curated image sets.
- Watch for text shortcuts: A large part of the models’ answers came from the indication and exam type rather than the pixels. Evaluations should include no-image controls like this one.
- Patients using chatbots: Uploading scan images to a general chatbot is not a substitute for a radiologist’s report, based on this evidence.
Find more evaluations like this on our Benchmarks & Evals studies page and in our Health & Medical Research section.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated image. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of Lichtle, Meddeb, Birot, Niang, Gastaud, Hanafi, Dujardin, Selomme, Cachard, Hamroun, Kuchcinski, Kim and Le Guellec (2026), “Diagnostic Accuracy and Navigation Behavior of Vision-Language Model Agents Reading Emergency CT and MRI Examinations,” arXiv:2611.23480v1 (cs.AI, submitted November 17, 2026), including the supplemental material. All numbers come from the paper. The chart was drawn by i10X from Table 1 and Table S4; the paper is licensed CC BY-NC-SA 4.0, so no original figures were reused. Agent and analysis code: github.com/bastienLeGuellec/radioagent. The study received no specific funding. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
This is one of the more useful medical AI papers in a while because it tests the job, not a quiz. The striking result is not the low accuracy of cheaper models, but how they fail: they look at the right slice, decide too early and then sound certain. That is a failure mode a benchmark score will not show you. Before AI agents read scans unsupervised, developers need to prove they can change their mind.






