Quick Take
- The problem: AI research agents write fluent, well-cited reports, but their conclusions are often more decisive than the cited sources support. The NYU authors call this “claim smoothing”.
- The measurement: Their SmoothScore pipeline scored claims from seven systems on four research benchmarks. Every system had at least 22% of claims materially smoothed on every benchmark; Gemini Deep Research reached 85% to 93%.
- The fix that works: Giving agents the full text of their cited sources plus calibration instructions cut materially smoothed claims by more than half. Either measure alone helped much less.
Deep research tools promise to read the literature for you. A new preprint by Tiasa Singha Roy, Manya Wadhwa and Greg Durrett of New York University asks whether their reports reflect what the sources actually say. The finding: the reports often flatten mixed or narrow evidence into confident, general conclusions, and the agents rarely open the pages they cite.
What the study shows
The team took 125 synthesis-heavy questions from ResearcherBench (65), ScholarQA-CS2 expert-written (25), DeepResearch Bench II (18) and ResearchRubrics (17). They generated reports with GPT-5.6 Luna and GPT-5.6 Sol plus web search, Claude Sonnet 5 and Claude Opus 5.5 plus web search, Gemini Deep Research, and the open agents DR Tulu and QUEST. Each key claim got a SmoothScore from 1 (faithful) to 5 (needs replacing); 3 or more counts as materially smoothed.
- Mean scores: GPT-5.6 Luna (1.66 to 1.95) and Sol (1.82 to 1.93) smoothed least. Gemini Deep Research was highest at 3.05 to 3.27; all other systems fell between 2.20 and 2.88.
- Severe smoothing: Gemini scored 4 or higher on 22% to 35% of its claims. Luna and Sol stayed at 1% to 4%.
- Claude in the middle: Sonnet 5 and Opus 5.5 produced about 1.4 to 2.5 times as many claims scoring 3 to 4 as Luna and Sol. Neither produced fundamentally misleading claims (score 5) at more than 1.1% on any dataset.
- Open agents: QUEST and DR Tulu had at least about 55% and 59% of claims at 3 or higher. QUEST produced up to about 7% score-5 claims despite using small evidence pools.
- Agents barely read: On ResearcherBench, Luna opened only 19% of the pages it cited and worked mostly from search snippets (median 585 characters). Sonnet 5 fetched 2% of its cited links, and 44 of its 65 reports opened no page at all.
| Condition (ResearcherBench) | Luna: claims ≥3 | Sonnet 5: claims ≥3 |
|---|---|---|
| Web search (original) | 29% | 61% |
| + calibration instructions | 26% | 44% |
| Full cited documents | 23% | 51% |
| Documents + instructions | 12% | 25% |
The paper names three recurring patterns. Unsupported composition stitches separately supported findings into a stronger joint claim. Scope expansion turns a result from one setup into a general rule. Premature closure uses words like “fundamentally resolves” or “definitively proves”. Comparative questions (“which approach is more promising?”) and broad overview questions invite the most smoothing.
Why it matters: calibration, not just citation accuracy
- A new failure class: Classic fact-checking asks whether a claim is supported by a citation. SmoothScore asks whether its strength and scope match all the evidence, so a claim can be “cited” and still misleading.
- Looking for counter-evidence: A support agent and a skeptic agent search the cited documents. Their findings are grouped by aspect so the judge weighs pro and contra side by side.
- Human check: On 73 claims, the GPT-5 judge matched one annotator exactly as often as the two annotators matched each other (56% vs. 58%) and was within one point 93% of the time.
- Evidence is useful on its own: Revising reports with the collected evidence raised expert rubric scores (Luna +4.4, Sonnet 5 +12.0, both p < .001) using about one-fifth to one-tenth of the tokens of the full documents.
- Open release: Code and data are on GitHub.
Method in brief
Pipeline: claim extraction, support and skeptic evidence passes over the cited documents, aspect induction, evidence assignment and a 1-to-5 judgment. All steps use GPT-5-mini except the final judge, which is GPT-5. Data: all 65 ResearcherBench and 25 ScholarQA-CS2 expert questions plus 18 of 66 DeepResearch Bench II and 17 of 101 ResearchRubrics tasks, kept because they demand open-ended synthesis. Statistics: 95% confidence intervals on mean scores, Spearman correlations between report features and smoothing.
What remains open
- OpenAI judging OpenAI: The judge is GPT-5, and the two GPT-5.6 systems scored best. Gemini 2.5 Pro and Claude Sonnet 4.5 as alternative judges kept the same ranking on the 73 validation claims, but they were less calibrated and the check was small.
- Small validation set: 73 claims, scored by two of the authors. Annotators agreed exactly on only 58%, which shows how subjective the task is.
- Limited engagement logs: Anthropic search results are encrypted, so Claude “opens” count only explicit web fetches. Gemini exposes no browsing steps at all.
- Naming and status: The paper calls the Claude model “Opus 5.5” in the text but “Claude Opus 5” in one appendix table. It is an unreviewed preprint without a separate limitations section.
Context: beyond hallucination checks
Earlier long-form factuality tools such as FActScore check whether individual claims are backed by sources, with yes-or-no answers. SmoothScore targets a subtler problem that matters as deep research becomes a default workflow: conclusions that are technically sourced but overstated. It adds to growing evidence that LLMs are shaky judges of research, as in our report on LLMs failing to spot tampered ML results. More evaluations are collected under Benchmarks & Evals.
What it means for teams using deep research agents
- Make agents read: Configure tools to fetch and read full sources, or pass documents into context, rather than relying on snippets.
- Ask for calibration: Instructions to hedge and state scope only worked well when the agent had the full documents. Use both.
- Rephrase comparative questions: “What does the evidence say about A and B?” invites less overclaiming than “Which is better?”
- Audit strong wording: Phrases like “definitively”, “fundamentally resolves” or “ensures” are red flags worth checking against the source.
Company coverage for the tested providers is on the OpenAI and Claude hubs.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of “Quantifying Claim Smoothing by LLM Research Agents” (arXiv:2610.35793v1, submitted October 26, 2026), including the appendices on judge validation and document engagement. All numbers come from the paper; i10X did not rerun the pipeline. The chart was redrawn by i10X from Table 1; the paper is licensed CC BY 4.0. AI tools assisted with research, drafting and the title illustration; the selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
Smoothing is arguably more dangerous than plain hallucination because it passes a citation check: every claim links to something real. The study’s most practical lesson is simple. Agents that do not read their sources cannot respect the caveats in them. Until deep research products expose how much of each source they actually read, readers should treat confident one-line conclusions in AI reports as hypotheses, not findings.
