Benchmarks & EvalsDeepSeek
Evidence check
  • Preprint
  • No code released
  • Not independently replicated

Study: Some Open Base Models Recite Benchmark Test Sets Word for Word

SII-GAIR researchers find ERNIE-5.5-Flash-Base reproduces 96% of HumanEval and 85% of OpenBookQA test items. Zeroing leaked items drops its average by 16.6 points.

Christopher Ort
By Christopher Ort · 6 min read
Share
A torn-open envelope with exam answer sheets lies on a table in a dimly lit server room
Image: AI-generated, i10X
On this page
  1. Quick Take
  2. In plain English
  3. What the study shows
  4. Why it matters: direct evidence, not a statistical hint
  5. Method in brief
  6. What remains open
  7. Context: from benchmaxxing to benchleaking
  8. What it means for model buyers and researchers
  9. About the analysis
  10. i10x Perspective
i10X Quick Take

Quick Take

  1. The finding: Given a few solved examples and one test question, some public base models keep writing out other questions from the same benchmark test set, with options and correct answers. For ERNIE-5.5-Flash-Base, 96.3% of HumanEval and 85.4% of OpenBookQA test items leaked this way.
  2. Why it matters: When leaked items are marked wrong, ERNIE-5.5-Flash-Base’s 20-benchmark average falls from 85.17 to 68.55 and it drops from first to last of four models. DeepSeek-V4-Flash-Base and Hy3-preview-Base lose only 1.5 and 1.3 points.
  3. What it means: Leaderboard scores of open base models cannot be taken at face value. The authors call for decontamination reports and contamination-adjusted scores as standard practice.

In plain English

AI models are ranked with standard tests, much like students sit standard exams. This study found that some AI models seem to have seen the exam papers, including the answers, while they were being trained. When the researchers showed one model a single test question, it answered and then wrote out the next test questions and their official answers from memory, even copying a spelling mistake from the original. Once the researchers stopped giving credit for those memorized questions, the model that looked best fell to last place.

What you should do: If you choose an AI model based on its test scores, those scores may partly measure memory, not ability.

What the study shows

Benchmark contamination is an old worry in AI research, but most evidence for it has been indirect: unusually low loss on test data, or a drop in accuracy on freshly written copies of a test. A team from SII-GAIR, Xinxu AI and GAIR led by Yiwei Qin, with Pengfei Liu as senior author, argues that recent releases show the problem in a sharper form. Their preprint, posted to arXiv on November 16, 2026, coins the term benchleaking for the case in which a benchmark’s own test examples enter training data, and separates it from benchmaxxing, where a developer tunes toward a benchmark without seeing its test set.

  • Base models recite unseen test items: In continuation runs where generation was allowed to continue past the answer, ERNIE-5.5-Flash-Base reproduced 96.3% of HumanEval problems with their reference implementations, 85.4% of OpenBookQA, 80.1% of GSM8K and 34.6% of GPQA-Main. MiMo-V2.5-Base reproduced 73.8% of HumanEval. In one example the model wrote out four HumanEval problems it was never shown, in test-set order, down to the typo “examble” in the original.
  • Other models leak far less: HumanEval leakage was 9.1% for DeepSeek-V4-Flash-Base and 6.7% for Hy3-preview-Base, and neither reproduced a single complete OpenBookQA or GPQA-Main item in this probe.
  • Scores shrink once leaks get zero credit: The authors define an Anti-Hack score: the benchmark score recomputed with every leaked item marked wrong and the test size unchanged. Averaged over 20 benchmarks, ERNIE-5.5-Flash-Base falls from 85.17 to 68.55, MiMo-V2.5-Base from 80.01 to 75.03, Hy3-preview-Base from 79.86 to 78.55 and DeepSeek-V4-Flash-Base from 77.95 to 76.48. ERNIE-5.5-Flash-Base’s HumanEval score drops from 100.00 to 3.66.
  • Signs in post-training agents too: Replicating two outside audits, the team found candidate statements of hidden Terminal-Bench 4.0 grading rules, absent from the instructions, from Kimi-K3.1-Pro on 14.3% of tasks during normal solving and 51.5% when asked to list its instructions, against 1.6% and 9.1% for Kimi-K3. On MaxxBench, where familiar DeepSWE coding tasks are slightly modified, pass rates fell by 13.3 points for DeepSeek-V4.1-Flash and 15.8 points for Kimi-K3.1-Pro.
Grouped bar chart of average benchmark scores before and after leaked test items are marked wrong: ERNIE-5.5-Flash-Base 85.17 to 68.55, MiMo-V2.5-Base 80.01 to 75.03, Hy3-preview-Base 79.86 to 78.55, DeepSeek-V4-Flash-Base 77.95 to 76.48.
i10X chart based on data from Qin, Xie, Li, Jiang, Fu, Mi, Guo and Liu (2026), arXiv:2611.21315, Section 3.2 and Figure 1. CC BY 4.0.
BenchmarkERNIE-5.5-Flash-BaseMiMo-V2.5-BaseHy3-preview-BaseDeepSeek-V4-Flash-Base
HumanEval96.3%73.8%6.7%9.1%
OpenBookQA85.4%0.0%0.0%0.0%
GSM8K80.1%4.4%5.1%0.8%
GPQA-Main34.6%0.0%0.0%0.0%
MBPP9.2%0.4%0.0%0.0%
Share of test examples reproduced in full (question, options or context, and reference answer). Selected rows of Table 4 in Qin et al. (2026), arXiv:2611.21315.

Why it matters: direct evidence, not a statistical hint

The strength of this paper is its strict matching rule. A test item counts as leaked only if the model writes out the complete question, all its options or context, and the reference answer, and only if none of it appeared anywhere in the input, including the few-shot examples. For code, a different solution that passes the tests does not count; the reference implementation must match. Case and whitespace are normalized, but paraphrases are not accepted. Because the model produces content it was never given, the authors argue that a full match is direct evidence that the test set was in the training data. The results in the main text use this strictest of three matching levels; an appendix reports the looser ones.

The Anti-Hack score is also deliberately simple. It does not try to guess whether a model could have solved a leaked item on its own; it just removes the credit, which gives a conservative lower bound on genuine capability and makes the ranking change easy to verify.

Method in brief

  • Models: Four public base models: ERNIE-5.5-Flash-Base (Baidu), DeepSeek-V4-Flash-Base (DeepSeek), Hy3-preview-Base (Tencent) and MiMo-V2.5-Base (Xiaomi). Post-training checks used Kimi-K3.1-Pro, Kimi-K3 and DeepSeek-V4.1-Flash.
  • Benchmarks: 20 benchmarks covering knowledge and commonsense (such as MMLU and OpenBookQA), reasoning and reading (BBH, DROP), math (GSM8K, MATH-500), code (HumanEval, HumanEval+, MBPP, MBPP+) and science (GPQA, SuperGPQA).
  • Leak detection: Separate continuation runs with stop strings and end-of-sequence stopping disabled, so the model writes until its output limit. Outputs are matched only against other items in the same test set. Scores come from standard evaluation runs with lm-evaluation-harness.
  • Post-training probes: Reproductions of the MaxxBench comparison on 89 shared tasks and of hidden-rule probes on 63 to 66 Terminal-Bench 4.0 tasks, with an automatic semantic judge deciding whether a stated rule appears only in the hidden tests.

What remains open

  • Four models only: The pre-training analysis covers four base models. It says nothing about models that were not tested, including closed models whose base versions are not public.
  • How the data got in is unknown: The paper shows that test items were memorized, not how they entered the training data or whether it was deliberate. It reports no response from the model developers.
  • Strict matching undercounts: Requiring full verbatim matches means partial or paraphrased memorization is not counted, so leakage may be higher than reported. WinoGrande matches are shown without leak markers because its twin-sentence design can produce matches by construction.
  • Agent findings rely on a judge: The hidden-rule rates come from an automatic semantic judge with one attempt per task, and the authors frame them as candidates and signs rather than proof.
  • No limitations section or funding statement: The preprint has neither. The project repository named in the paper, GAIR-NLP/Benchleaking on GitHub, did not resolve publicly when we checked on November 17, 2026. The paper has not been peer-reviewed.

Context: from benchmaxxing to benchleaking

Earlier detection methods looked for low loss on test sets (Wei et al., 2023), completion of a test question from its first half (Golchin and Surdeanu, 2024; Sainz et al., 2023) or accuracy drops on fresh replicas (Zhang et al., 2024). The authors note these signals can have other explanations. This paper builds on two recent audits, MaxxBench by Lo and Lo (2026) and an R40 report by Schaeffer and Räuker (2026) on Kimi K3.1 Pro recalling hidden test rules, and adds direct reproduction of test items by base models. It fits a pattern we have covered in other evaluation studies, such as the ExpertCrowd-SWE coding benchmark, where newly written expert tasks are used to measure what models can really do.

What it means for model buyers and researchers

  • Teams choosing a model: Treat headline benchmark numbers as a starting point. Test candidates on your own private tasks, which cannot have leaked into training data.
  • Researchers building on open base models: A base model with leaked test sets is not a clean starting point. Scores of fine-tuned versions on the same benchmarks inherit the contamination.
  • Model developers: Publish decontamination reports and contamination-adjusted scores. DeepSeek‘s base model shows that strong results without heavy leakage are possible: it kept almost all of its score.

More evaluation research is collected on our Benchmarks & Evals studies page.

About the analysis

Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated image. Read our AI Use Policy. Spotted an error? Request a correction.

This article is based on the full text of Qin, Xie, Li, Jiang, Fu, Mi, Guo and Liu (2026), “Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month, from post-training down to pre-training base models,” arXiv:2611.21315v1 (cs.AI, submitted November 16, 2026). All numbers come from the paper. The chart was redrawn by i10X from Section 3.2 and Figure 1; the paper is published under a CC BY 4.0 license. The paper names a GitHub project, GAIR-NLP/Benchleaking. No funding or conflict-of-interest statement is included. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.

i10x Perspective

A model that recites the next exam question, typo included, is not reasoning. This paper is a useful reminder that public leaderboards are only as trustworthy as the training data behind them, and that nobody outside the labs can audit that data. Until contamination reports become routine, private evaluations on your own tasks remain the only scores worth betting a product on.

Christopher Ort

Christopher Ort

Christopher Ort handles SEO, AI agent workflows and PR & partnerships at i10x.ai, the all-in-one AI workspace built by i10X Global Pte. Ltd. in Singapore. He is a graduate of Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). He is the author of “The Style Penalty”, a study published on SSRN that analyses 1,576 evaluation decisions by four frontier AI models to show how AI-written CVs fare in automated hiring screens; the research has been covered by heise online and t3n.

Timeline

What happened before

  1. Release

    DeepSeek-V4.1-Flash-1017

    1M context · open weights

    api-docs.deepseek.com (source, opens in a new tab)
  2. Release

    DeepSeek-V4.1-Flash

    1M context · $0.3 / $1.2 per 1M tokens · Superseded 17 Oct 2026 by V4.1-Flash-1017

    api-docs.deepseek.com (source, opens in a new tab)
  3. Release

    DeepSeek-V4-Pro (GA)

    1M context · $1.32 / $3.96 per 1M tokens

    api-docs.deepseek.com (source, opens in a new tab)
  4. Release

    DeepSeek-V4-Flash

    Retired Oct 2026 (served by V4.1-Flash-1017)

    api-docs.deepseek.com (source, opens in a new tab)
  5. Funding

    DeepSeek closes first external round of over $7B

    First external round · ¥50B+ at $50B valuation

    Per The Information via Reuters: >¥50B (~$7.4B) at a post-money valuation above $50B, with an unusual deal structure.

    reuters.com (source, opens in a new tab)
Full DeepSeek timeline ›

Editor's picks

  1. Genesys Launches Standalone AI Agents and Arena, With Pricing Based on How Hard Each Task Is
  2. Seedance 2.5 4K Reaches Developers: 10-Bit Video up to 30 Seconds With 480p Draft Mode
  3. Perplexity Computer Adds Multiagent Mode, Where Up to 16 Agents on Rival Models Check Each Other
  4. Gemini Can Now Make Calls, Pay Bills and Shop, Free in Six Countries With Gemini 4 Hydrogen
  5. Hinton on Firing Line: AI Is Making ‘Alien Beings’ and Self-Regulation Is ‘Absurd’