Quick Take
- Harsh: Asked to review 200 papers already published in 2026 in journals such as Nature, Science and Cell, four LLMs recommended major revision or rejection in 605 of 800 reviews (75.6%).
- Inconsistent: The rate ranged from 17.5% for Gemini 3.7 Flash to 100% for GPT-5.6 Sol, and one framing sentence saying the paper was a revision raised Gemini’s accept rate from 15.5% to 68.5%.
- Blind to real flaws: When one serious flaw was planted in each paper, the models caught it in only 17.0% to 60.5% of cases, and their recommendations barely changed.
Journals and conferences are drowning in submissions, and LLM reviewers are increasingly proposed as relief. A preprint from Bastien Le Guellec and colleagues at CHU Lille and the Technical University of Munich tests them against the closest thing science has to a gold standard: articles that have just passed peer review at top journals. The models were not told the papers were already published. Their verdicts say more about the model and the prompt than about the paper.
What the study shows
| Reviewer model (via OpenRouter) | Major revision or reject (published papers) | Caught the planted flaw |
|---|---|---|
| GPT-5.6 Sol | 200/200 (100%) | 121/200 (60.5%) |
| GLM-5.3 Flash | 180/200 (90.0%) | 88/200 (44.0%) |
| DeepSeek v4 Flash | 190/200 (95.0%) | 34/200 (17.0%) |
| Gemini 3.7 Flash | 35/200 (17.5%) | 36/200 (18.0%) |
- Almost no agreement: Only 21 of 200 papers (10.5%) got the same decision from all four models, and every one of those was a major revision. No paper was unanimously accepted.
- Prestige made no difference: Decisions did not differ by journal (p = 0.24). Papers from Nature, Science and Cell scored 1.74 on the four-point severity scale, Scientific Reports 1.98 (p = 0.081).
- One sentence of framing: Telling the models the paper had already been through one round of review made all four more lenient. Mean severity fell from 1.77 to 1.26; 392 of 800 reviews became milder and 6 harsher. Accept rose from 3.9% to 17.8%, reject fell from 5.4% to 0.9%.
- Flaws found, verdict unchanged: Planting a severe statistical or methodological flaw raised mean severity by just 0.02 of a category. Even where the flaw was caught, severity rose only 0.07 and the recommendation stayed the same in 84.2% of reviews.
- One clear winner on quality: Two outside judges named GPT-5.6 Sol’s review the best of four in 94.5% (Grok 4.6) and 75.5% (MiniMax M3) of papers. Reviews from Gemini 3.7 Flash and DeepSeek v4 Flash were never picked.
Why it matters: specificity and sensitivity in one design
- Both error types measured: Published papers test whether models wrongly reject sound work (low specificity); the planted flaws test whether they miss real problems (low sensitivity). The models fail on both counts.
- Realistic, broad corpus: 200 articles, ten from each of 20 general-science, life-science and clinical journals, all published in 2026 under CC BY 4.0, with at least 2,000 words of body text.
- Controlled framing test: The two prompts differed only in their first sentence, which isolates how much the verdict depends on context rather than content.
- Off-panel judging: Flaws were planted by Grok 4.6, detection was judged by Claude Opus 5, and review quality by Grok 4.6 and MiniMax M3, none of which were reviewers. Partial detections were reported separately: counting them as catches would raise the range to 29.5% to 84.0%.
Method in brief
Reviewers: GLM-5.3 Flash, GPT-5.6 Sol, Gemini 3.7 Flash and DeepSeek v4 Flash via the OpenRouter API, chosen for a top-50 leaderboard rank and a price below $5 per million output tokens; two proprietary, two open-weight. Protocol: the prompt named the paper’s real target journal and withheld that it was published; each model reviewed each paper twice, as a first submission and as a revision. Experiment 2: one flaw per paper, statistical (for example clustered data treated as independent) or methodological, inserted as a targeted text substitution and designed to warrant major revision or rejection on its own. Experiment 3: the four reviews per paper, relabelled and shuffled, rated by two judge models for best review, relevance, constructiveness and nitpicking. Decisions were analysed as a four-point severity scale with non-parametric tests.
What remains open
- Published is not the same as sound: The design assumes the journals’ editorial work is a reasonable basis of trust. Some of the models’ criticisms may be valid; no human expert assessed the reviews.
- Cheaper models only: For cost and carbon reasons the authors did not test the largest or most expensive models, so frontier reviewers may perform differently.
- LLMs judging LLMs: Flaw detection and review quality were scored by other models. The two quality judges agreed on the best review in 79.0% of papers, but with a low Cohen’s κ of 0.23.
- Possible contamination and no data yet: Some articles may have appeared earlier as preprints and been seen in training. The data is to be deposited in a public repository before journal submission; it is not available yet. The preprint is not peer reviewed.
Context: AI review is already here
The paper notes that AI review is no longer hypothetical: a 2025 Frontiers survey found that 53% of reviewers already use AI, every main-track submission to AAAI-26 (22,977 papers) received an AI-generated review, and NEJM AI offers an accelerated human-plus-AI review track. Kim and colleagues have already reviewed the quality of AI reviews of Nature-family papers; this study focuses on the recommendations themselves. It joins i10X coverage of AI’s growing role in research, from AI-generated mathematics manuscripts to the risk of scientific monocultures.
What it means for editors, reviewers and authors
- Editors: Do not treat an LLM’s accept or reject recommendation as a signal. It depends heavily on which model is used and how the request is framed.
- Reviewers: LLM reports can surface concerns, especially ones several models raise independently, but a clean LLM review is no evidence that a paper has no serious flaw.
- Authors: A harsh “major revision” from a chatbot before submission says little; the same text framed differently may get a much milder verdict.
More on the providers behind the tested reviewers: OpenAI, Google and DeepSeek, or browse society and economy research.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint “Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals” (arXiv:2610.31414v1, submitted October 22, 2026). All numbers come from the paper; i10X did not rerun the experiments. The authors report no dedicated funding and no competing interests, and state that an AI agent (Claude) carried out corpus retrieval, experiments, statistical analysis, figures and drafting under the corresponding author’s direction and review. The paper is licensed CC BY-NC-SA 4.0; no original figures are reproduced, and the chart was redrawn by i10X from the reported numbers.
i10x Perspective
The most useful finding here is not that LLMs are harsh, but that their verdicts are arbitrary: the same paper gets accept or major revision depending on the model and one sentence of framing, while a planted fatal flaw barely moves the needle. That makes the recommendation the least trustworthy part of an AI review. The concerns themselves, especially those several models agree on, look more promising. Publishers rolling out AI review should design around that split, and test with planted flaws before trusting any model’s verdict.
