Quick Take
- The finding: Given a few solved examples and one test question, some public base models keep writing out other questions from the same benchmark test set, with options and correct answers. For ERNIE-5.5-Flash-Base, 96.3% of HumanEval and 85.4% of OpenBookQA test items leaked this way.
- Why it matters: When leaked items are marked wrong, ERNIE-5.5-Flash-Base’s 20-benchmark average falls from 85.17 to 68.55 and it drops from first to last of four models. DeepSeek-V4-Flash-Base and Hy3-preview-Base lose only 1.5 and 1.3 points.
- What it means: Leaderboard scores of open base models cannot be taken at face value. The authors call for decontamination reports and contamination-adjusted scores as standard practice.
In plain English
AI models are ranked with standard tests, much like students sit standard exams. This study found that some AI models seem to have seen the exam papers, including the answers, while they were being trained. When the researchers showed one model a single test question, it answered and then wrote out the next test questions and their official answers from memory, even copying a spelling mistake from the original. Once the researchers stopped giving credit for those memorized questions, the model that looked best fell to last place.
What you should do: If you choose an AI model based on its test scores, those scores may partly measure memory, not ability.
What the study shows
Benchmark contamination is an old worry in AI research, but most evidence for it has been indirect: unusually low loss on test data, or a drop in accuracy on freshly written copies of a test. A team from SII-GAIR, Xinxu AI and GAIR led by Yiwei Qin, with Pengfei Liu as senior author, argues that recent releases show the problem in a sharper form. Their preprint, posted to arXiv on November 16, 2026, coins the term benchleaking for the case in which a benchmark’s own test examples enter training data, and separates it from benchmaxxing, where a developer tunes toward a benchmark without seeing its test set.
- Base models recite unseen test items: In continuation runs where generation was allowed to continue past the answer, ERNIE-5.5-Flash-Base reproduced 96.3% of HumanEval problems with their reference implementations, 85.4% of OpenBookQA, 80.1% of GSM8K and 34.6% of GPQA-Main. MiMo-V2.5-Base reproduced 73.8% of HumanEval. In one example the model wrote out four HumanEval problems it was never shown, in test-set order, down to the typo “examble” in the original.
- Other models leak far less: HumanEval leakage was 9.1% for DeepSeek-V4-Flash-Base and 6.7% for Hy3-preview-Base, and neither reproduced a single complete OpenBookQA or GPQA-Main item in this probe.
- Scores shrink once leaks get zero credit: The authors define an Anti-Hack score: the benchmark score recomputed with every leaked item marked wrong and the test size unchanged. Averaged over 20 benchmarks, ERNIE-5.5-Flash-Base falls from 85.17 to 68.55, MiMo-V2.5-Base from 80.01 to 75.03, Hy3-preview-Base from 79.86 to 78.55 and DeepSeek-V4-Flash-Base from 77.95 to 76.48. ERNIE-5.5-Flash-Base’s HumanEval score drops from 100.00 to 3.66.
- Signs in post-training agents too: Replicating two outside audits, the team found candidate statements of hidden Terminal-Bench 4.0 grading rules, absent from the instructions, from Kimi-K3.1-Pro on 14.3% of tasks during normal solving and 51.5% when asked to list its instructions, against 1.6% and 9.1% for Kimi-K3. On MaxxBench, where familiar DeepSWE coding tasks are slightly modified, pass rates fell by 13.3 points for DeepSeek-V4.1-Flash and 15.8 points for Kimi-K3.1-Pro.

| Benchmark | ERNIE-5.5-Flash-Base | MiMo-V2.5-Base | Hy3-preview-Base | DeepSeek-V4-Flash-Base |
|---|---|---|---|---|
| HumanEval | 96.3% | 73.8% | 6.7% | 9.1% |
| OpenBookQA | 85.4% | 0.0% | 0.0% | 0.0% |
| GSM8K | 80.1% | 4.4% | 5.1% | 0.8% |
| GPQA-Main | 34.6% | 0.0% | 0.0% | 0.0% |
| MBPP | 9.2% | 0.4% | 0.0% | 0.0% |
Why it matters: direct evidence, not a statistical hint
The strength of this paper is its strict matching rule. A test item counts as leaked only if the model writes out the complete question, all its options or context, and the reference answer, and only if none of it appeared anywhere in the input, including the few-shot examples. For code, a different solution that passes the tests does not count; the reference implementation must match. Case and whitespace are normalized, but paraphrases are not accepted. Because the model produces content it was never given, the authors argue that a full match is direct evidence that the test set was in the training data. The results in the main text use this strictest of three matching levels; an appendix reports the looser ones.
The Anti-Hack score is also deliberately simple. It does not try to guess whether a model could have solved a leaked item on its own; it just removes the credit, which gives a conservative lower bound on genuine capability and makes the ranking change easy to verify.
Method in brief
- Models: Four public base models: ERNIE-5.5-Flash-Base (Baidu), DeepSeek-V4-Flash-Base (DeepSeek), Hy3-preview-Base (Tencent) and MiMo-V2.5-Base (Xiaomi). Post-training checks used Kimi-K3.1-Pro, Kimi-K3 and DeepSeek-V4.1-Flash.
- Benchmarks: 20 benchmarks covering knowledge and commonsense (such as MMLU and OpenBookQA), reasoning and reading (BBH, DROP), math (GSM8K, MATH-500), code (HumanEval, HumanEval+, MBPP, MBPP+) and science (GPQA, SuperGPQA).
- Leak detection: Separate continuation runs with stop strings and end-of-sequence stopping disabled, so the model writes until its output limit. Outputs are matched only against other items in the same test set. Scores come from standard evaluation runs with lm-evaluation-harness.
- Post-training probes: Reproductions of the MaxxBench comparison on 89 shared tasks and of hidden-rule probes on 63 to 66 Terminal-Bench 4.0 tasks, with an automatic semantic judge deciding whether a stated rule appears only in the hidden tests.
What remains open
- Four models only: The pre-training analysis covers four base models. It says nothing about models that were not tested, including closed models whose base versions are not public.
- How the data got in is unknown: The paper shows that test items were memorized, not how they entered the training data or whether it was deliberate. It reports no response from the model developers.
- Strict matching undercounts: Requiring full verbatim matches means partial or paraphrased memorization is not counted, so leakage may be higher than reported. WinoGrande matches are shown without leak markers because its twin-sentence design can produce matches by construction.
- Agent findings rely on a judge: The hidden-rule rates come from an automatic semantic judge with one attempt per task, and the authors frame them as candidates and signs rather than proof.
- No limitations section or funding statement: The preprint has neither. The project repository named in the paper, GAIR-NLP/Benchleaking on GitHub, did not resolve publicly when we checked on November 17, 2026. The paper has not been peer-reviewed.
Context: from benchmaxxing to benchleaking
Earlier detection methods looked for low loss on test sets (Wei et al., 2023), completion of a test question from its first half (Golchin and Surdeanu, 2024; Sainz et al., 2023) or accuracy drops on fresh replicas (Zhang et al., 2024). The authors note these signals can have other explanations. This paper builds on two recent audits, MaxxBench by Lo and Lo (2026) and an R40 report by Schaeffer and Räuker (2026) on Kimi K3.1 Pro recalling hidden test rules, and adds direct reproduction of test items by base models. It fits a pattern we have covered in other evaluation studies, such as the ExpertCrowd-SWE coding benchmark, where newly written expert tasks are used to measure what models can really do.
What it means for model buyers and researchers
- Teams choosing a model: Treat headline benchmark numbers as a starting point. Test candidates on your own private tasks, which cannot have leaked into training data.
- Researchers building on open base models: A base model with leaked test sets is not a clean starting point. Scores of fine-tuned versions on the same benchmarks inherit the contamination.
- Model developers: Publish decontamination reports and contamination-adjusted scores. DeepSeek‘s base model shows that strong results without heavy leakage are possible: it kept almost all of its score.
More evaluation research is collected on our Benchmarks & Evals studies page.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated image. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of Qin, Xie, Li, Jiang, Fu, Mi, Guo and Liu (2026), “Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month, from post-training down to pre-training base models,” arXiv:2611.21315v1 (cs.AI, submitted November 16, 2026). All numbers come from the paper. The chart was redrawn by i10X from Section 3.2 and Figure 1; the paper is published under a CC BY 4.0 license. The paper names a GitHub project, GAIR-NLP/Benchleaking. No funding or conflict-of-interest statement is included. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
A model that recites the next exam question, typo included, is not reasoning. This paper is a useful reminder that public leaderboards are only as trustworthy as the training data behind them, and that nobody outside the labs can audit that data. Until contamination reports become routine, private evaluations on your own tasks remain the only scores worth betting a product on.






