{"id":6607,"date":"2026-10-26T08:12:26","date_gmt":"2026-10-26T08:12:26","guid":{"rendered":"https:\/\/i10xnews.kinsta.cloud\/?p=6607"},"modified":"2026-10-26T08:12:26","modified_gmt":"2026-10-26T08:12:26","slug":"tampered-ml-benchmark-llm-judgment-ml-research","status":"publish","type":"post","link":"https:\/\/i10x.ai\/news\/tampered-ml-benchmark-llm-judgment-ml-research","title":{"rendered":"Tampered ML: Even the Best AI Models Miss One in Five Planted Research Flaws"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">\u26a1 Quick Take<\/h2>\n\n\n<ul class=\"wp-block-list\"><li><strong>The test:<\/strong> Tampered ML pairs 1,858 experiment designs from top ML papers with a copy that has one planted methodological flaw. Models must pick the sound version in both presentation orders.<\/li><li><strong>The result:<\/strong> Frontier models beat chance but are far from reliable. The best, Claude Opus 5 and Claude Fable 5, reach about 78 to 79% in both orders; many models fall well below that.<\/li><li><strong>The weakness:<\/strong> Models change their verdict when the order is swapped, and often prefer the flawed design because it sounds more specific, transparent or conventional.<\/li><\/ul>\n\n\n<p class=\"wp-block-paragraph\">If AI systems are to help run AI research, including safety research, they have to tell a good experiment from a subtly broken one before anyone spends compute on it. A new paper by Soroush Ebadian (Anthropic Fellows Program), Parand A. Alamdari (University of Toronto), and Joe Benton and Hailey Joren (Anthropic), accepted at EMNLP 2026, builds a benchmark for exactly that skill. Its verdict: today&#8217;s models notice many flaws, but cannot yet be trusted to weigh them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd2c What the study shows<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The authors started from oral and spotlight papers at NeurIPS, ICML and ICLR from 2024 to 2026 (981 papers in the leaderboard set). A language-model pipeline extracted one experiment per item, rewrote it as a self-contained proposal, then planted a single flaw by a minimal edit, changing a median of only 2.0% of the text. Several automated gates checked that the original stays faithful to the paper and that the edit really changes the experiment. Each model sees the pair twice, once in each order, and only gets credit if it picks the original both times; random guessing scores 25%.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1845\" height=\"745\" src=\"https:\/\/i10xnews.kinsta.cloud\/wp-content\/uploads\/2026\/10\/tampered-ml-figure1-pairwise-accuracy.webp\" alt=\"Bar chart from the paper showing pairwise accuracy of 22 models on 1,858 tampered experiments. Claude Fable 5 and Claude Opus 5 lead at about 78 to 79 percent both-orders accuracy, followed by Opus 4.8, Gemini 3.1 Pro and GPT-5.5; Qwen3-30B ranks lowest.\" class=\"wp-image-6606\" srcset=\"https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-figure1-pairwise-accuracy.webp 1845w, https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-figure1-pairwise-accuracy-300x121.webp 300w, https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-figure1-pairwise-accuracy-1024x413.webp 1024w, https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-figure1-pairwise-accuracy-768x310.webp 768w, https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-figure1-pairwise-accuracy-1536x620.webp 1536w\" sizes=\"auto, (max-width: 1845px) 100vw, 1845px\" \/><figcaption class=\"wp-element-caption\">Dark bars: correct in both orders; light bars: per-order accuracy. The gap shows position sensitivity. Figure 1 from Ebadian, Alamdari, Benton &amp; Joren (2026), arXiv:2610.32114, CC BY 4.0.<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Measure<\/th><th>Result<\/th><\/tr><\/thead><tbody><tr><td>Best both-orders accuracy<\/td><td><strong>77.9%<\/strong> (Claude Opus 5); Claude Fable 5 79.6% on the items it answers, 73.5% counting its 7.8% refusals as errors<\/td><\/tr><tr><td>Best per-order accuracy<\/td><td>86 to 88% (Opus 5, Fable 5)<\/td><\/tr><tr><td>Order effect, Claude Opus 4.6<\/td><td>about 66% per order vs. 43% in both orders<\/td><\/tr><tr><td>Rating proposals one at a time (0 to 100)<\/td><td>4 to 19 points lower than pairwise; best Opus 5 at 68.8%, Gemini 3.1 Pro 67.6%<\/td><\/tr><tr><td>Estimated share of valid items (human check)<\/td><td>86% (95% CI 81 to 91%)<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Key results. Chance level is 25% for both-orders and 50% for per-order and pointwise accuracy. Source: Ebadian et al. (2026), Sections 3 and 4.<\/figcaption><\/figure>\n\n\n<ul class=\"wp-block-list\"><li><strong>Right flaw, wrong verdict:<\/strong> When a model was right in only one order, its reasoning usually named the planted flaw (above 86% for the stronger models). Order bias, not ignorance, explains many misses.<\/li><li><strong>Order shifts confidence:<\/strong> Even when correct in both orders, several models, especially the Opus models and GPT-5.5, were less confident and thought longer when the original came second.<\/li><li><strong>Typical errors:<\/strong> The authors catalog seven failure modes, including treating common practice as proof of validity, reading added detail as rigor, echoing the flawed proposal&#8217;s own justification, contradicting themselves across orders, and &#8220;recalling&#8221; the source paper incorrectly.<\/li><li><strong>Hints help, compute helps less:<\/strong> On a hard subset of 896 items, telling Opus 4.7 and Gemini 3.1 Pro that one design is flawed, and especially naming the flaw category, recovered many errors. More reasoning effort raised accuracy within each model, with diminishing returns; Opus 5 at its lowest effort already matched or beat most other models&#8217; best settings.<\/li><\/ul>\n\n\n<h2 class=\"wp-block-heading\">\u2705 Why it matters: judging experiments is an oversight skill<\/h2>\n\n\n<ul class=\"wp-block-list\"><li><strong>Clear ground truth:<\/strong> Comparing two independent research proposals rarely has a right answer. Minimal edits make the label clean: the tampered design only needs to be worse along one dimension.<\/li><li><strong>Order-robust scoring:<\/strong> Requiring the right answer in both orders exposes position bias that a single-order score would hide.<\/li><li><strong>Human validation:<\/strong> Late-stage ML PhD students and PhD graduates reviewed 210 items, deliberately focused on items strong models missed; 166 were judged valid. Post-stratified, that gives the 86% estimate.<\/li><li><strong>Built for reuse:<\/strong> A public subset is on <a href=\"https:\/\/github.com\/tamperedml\/tamperedml\" target=\"_blank\" rel=\"noopener\">GitHub<\/a> with a canary string against training contamination; the rest is available to researchers on request.<\/li><\/ul>\n\n\n<h2 class=\"wp-block-heading\">\ud83e\uddea Method in brief<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Source:<\/strong> 981 oral and spotlight papers with LaTeX source from NeurIPS, ICML and ICLR 2024 to 2026. <strong>Construction:<\/strong> Claude Opus 4.6\/4.7 and Sonnet 4.6 extract a structured fact set, write the original proposal, plan and apply one flaw, and pass gates for faithfulness, real divergence from the paper, experimental non-equivalence, self-containedness and absence of giveaway cues. A difficulty filter removed items a panel of models already solved, leaving 1,858. <strong>Flaw types:<\/strong> confounded comparison (20.5%), blocked mechanism (20.3%), measurement mismatch (18.0%), unfair comparison (17.1%) and four smaller categories including data leakage (2.8%). <strong>Models:<\/strong> 22 models from Anthropic, OpenAI, Google, xAI, DeepSeek, Moonshot, Z.ai, Qwen and MiniMax, mostly at high reasoning effort.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u2753 What remains open<\/h2>\n\n\n<ul class=\"wp-block-list\"><li><strong>Models built the test:<\/strong> The pipeline uses language models to generate and check items, so model errors can shape the benchmark. About one item in seven is estimated not to be clearly valid.<\/li><li><strong>Thin human check:<\/strong> Each reviewed item had a single validator, and only 128 of the reviewed items are in the 1,858-item leaderboard set.<\/li><li><strong>Easier than reality:<\/strong> Real proposals have several interacting weaknesses and no matched counterpart to compare against. The authors call their accuracies optimistic estimates, and pointwise scoring already shows a lower ceiling.<\/li><li><strong>Judging is not designing:<\/strong> The benchmark measures whether models recognize flaws, not whether they avoid them when they write experiments themselves. Construction pipeline code is not released.<\/li><\/ul>\n\n\n<h2 class=\"wp-block-heading\">\ud83e\udded Context: from finding errors to weighing them<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Earlier benchmarks such as FLAWS and PaperAudit-Bench plant errors in full papers and ask models to locate them; MLE-Bench, RE-Bench and PaperBench test agents as executors of ML research. Tampered ML targets a narrower skill: deciding which of two near-identical designs supports its claim. It fits a growing body of evidence that LLM reviewers sound convincing while missing what matters, covered in i10X&#8217;s <a href=\"\/news\/category\/benchmarks-evals\">Benchmarks &amp; Evals<\/a> section.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udee0\ufe0f What it means for research teams using AI reviewers<\/h2>\n\n\n<ul class=\"wp-block-list\"><li><strong>Swap the order:<\/strong> When asking a model to compare two designs, run both orders and treat disagreement as a warning sign.<\/li><li><strong>Ask for a defect, not a winner:<\/strong> Prompts that say a flaw exists and suggest where to look recovered many errors in the study.<\/li><li><strong>Distrust polish:<\/strong> Extra specificity and &#8220;standard practice&#8221; language are exactly what fooled models. Keep a human in the loop for go\/no-go decisions on experiments.<\/li><\/ul>\n\n\n<p class=\"wp-block-paragraph\">More on the lab behind the benchmark and its leading models is on the <a href=\"\/news\/company\/anthropic\">Anthropic<\/a> hub and the <a href=\"\/news\/product\/claude\">Claude<\/a> product page.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u270d\ufe0f About the analysis<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This article is based on the full text of &#8220;Tampered ML: Evaluating LLM Judgment of Machine Learning Research&#8221; (arXiv:2610.32114v1, submitted October 23, 2026; EMNLP 2026 Main), including the limitations and the human-validation appendix. All numbers come from the paper; i10X did not rerun the evaluation. Three of the four authors are affiliated with Anthropic or its Fellows Program, and Anthropic models top the leaderboard; the paper contains no separate funding or competing-interest statement. The paper is licensed CC BY 4.0, and Figure 1 is reproduced under that license. AI tools assisted with research, drafting and the title illustration; the selection, fact-checking and final editing were done by the i10X editorial team.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd2d i10x Perspective<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The most telling result is not the leaderboard but the pattern of errors. Models often see the flaw and then talk themselves out of it, rewarding detail and convention over validity. That is a very human failure, and a dangerous one to automate: a reviewer that explains itself fluently is easy to trust. Labs that want AI to propose and filter experiments, including for safety research, will need order-robust checks and explicit flaw hunting built into the loop. It is also worth noting who sits at the top: an Anthropic-led benchmark ranked Anthropic models first, so independent replication on the public subset would add weight.<\/p>\n\n","protected":false},"excerpt":{"rendered":"<p>A new EMNLP 2026 benchmark pairs 1,858 ML experiment designs with flawed twins. Claude Opus 5 leads at 77.9%, but models flip verdicts when the order changes and favor polished flaws.<\/p>\n","protected":false},"author":2,"featured_media":6605,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"paper_title":"Tampered ML: Evaluating LLM Judgment of Machine Learning Research","paper_url":"https:\/\/arxiv.org\/abs\/2610.32114","pdf_url":"https:\/\/arxiv.org\/pdf\/2610.32114v1","doi":"","arxiv_id":"2610.32114","authors":"Soroush Ebadian, Parand A. Alamdari, Joe Benton, Hailey Joren","institutions":"Anthropic Fellows Program; University of Toronto; Anthropic","paper_date":"2026-10-23","venue":"EMNLP 2026 Main (arXiv cs.CL, version 1)","peer_reviewed":true,"code_url":"https:\/\/github.com\/tamperedml\/tamperedml","data_url":"https:\/\/github.com\/tamperedml\/tamperedml","funding_coi":"No funding or competing-interest statement; three of four authors affiliated with Anthropic or the Anthropic Fellows Program.","figure_license":"Original Figure 1 reused under CC BY 4.0","paper_license":"CC BY 4.0","replication":"partial","study_models":"22 models incl. Claude Fable 5, Opus 5, Opus 4.5-4.8, Sonnet 4.5\/4.6, Haiku 4.5; GPT-5.2, 5.4, 5.5; Gemini 3.1 Pro, 3 Flash; Grok 4.3; DeepSeek V4, V3.1; Kimi K2.6; GLM-5.1; Qwen3-235B, Qwen3-30B; MiniMax M2.5","study_sample":"1,858 original\/tampered experiment-design pairs from 981 oral and spotlight papers (NeurIPS, ICML, ICLR 2024-2026); 210 items human-validated","footnotes":"","_yoast_wpseo_primary_category":""},"categories":[53,52],"tags":[],"companies":[39,37,38],"products":[51,49],"class_list":["post-6607","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-benchmarks-evals","category-studies","company-anthropic","company-google","company-open-ai","product-claude","product-gemini"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Tampered ML Benchmark: AI Misses Research Flaws - i10X News<\/title>\n<meta name=\"description\" content=\"The Tampered ML benchmark tests whether AI models spot planted flaws in 1,858 ML experiments. Claude Opus 5 leads at 77.9%, but order bias trips up models.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Tampered ML Benchmark: AI Misses Research Flaws - i10X News\" \/>\n<meta property=\"og:description\" content=\"The Tampered ML benchmark tests whether AI models spot planted flaws in 1,858 ML experiments. Claude Opus 5 leads at 77.9%, but order bias trips up models.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research\" \/>\n<meta property=\"og:site_name\" content=\"i10X News\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-26T08:12:26+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/news.i10x.ai\/wp-content\/uploads\/2026\/10\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"1344\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Christopher Ort\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@i10x_ai\" \/>\n<meta name=\"twitter:site\" content=\"@i10x_ai\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Christopher Ort\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"NewsArticle\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research\"},\"author\":{\"name\":\"Christopher Ort\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#\\\/schema\\\/person\\\/d410bb7ee72febefca2c9c369939219a\"},\"headline\":\"Tampered ML: Even the Best AI Models Miss One in Five Planted Research Flaws\",\"datePublished\":\"2026-10-26T08:12:26+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research\"},\"wordCount\":1258,\"publisher\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i10x.ai\\\/news\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp\",\"articleSection\":[\"Benchmarks &amp; Evals\",\"Studies\"],\"inLanguage\":\"en-US\",\"dateModified\":\"2026-10-26T08:12:26+00:00\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research\",\"url\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research\",\"name\":\"Tampered ML Benchmark: AI Misses Research Flaws - i10X News\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i10x.ai\\\/news\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp\",\"datePublished\":\"2026-10-26T08:12:26+00:00\",\"description\":\"The Tampered ML benchmark tests whether AI models spot planted flaws in 1,858 ML experiments. Claude Opus 5 leads at 77.9%, but order bias trips up models.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research#primaryimage\",\"url\":\"https:\\\/\\\/i10x.ai\\\/news\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp\",\"contentUrl\":\"https:\\\/\\\/i10x.ai\\\/news\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp\",\"width\":1344,\"height\":768,\"caption\":\"Illustration: i10X\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/tampered-ml-benchmark-llm-judgment-ml-research#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/news.i10x.ai\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Tampered ML: Even the Best AI Models Miss One in Five Planted Research Flaws\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#website\",\"url\":\"https:\\\/\\\/news.i10x.ai\\\/\",\"name\":\"i10X News\",\"description\":\"Stay updated with the latest news and announcements from i10X.\",\"publisher\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/news.i10x.ai\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#organization\",\"name\":\"i10X\",\"url\":\"https:\\\/\\\/news.i10x.ai\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/i10xnews.kinsta.cloud\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/i10x-logo-180.png\",\"contentUrl\":\"https:\\\/\\\/i10xnews.kinsta.cloud\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/i10x-logo-180.png\",\"width\":180,\"height\":180,\"caption\":\"i10X\"},\"image\":{\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/x.com\\\/i10x_ai\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/i10x-ai\",\"https:\\\/\\\/instagram.com\\\/i10x.ai\",\"https:\\\/\\\/www.youtube.com\\\/@i10X_ai_official\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/news.i10x.ai\\\/#\\\/schema\\\/person\\\/d410bb7ee72febefca2c9c369939219a\",\"name\":\"Christopher Ort\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/i10x.ai\\\/news\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/christopher-ort-192.webp\",\"url\":\"https:\\\/\\\/i10x.ai\\\/news\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/christopher-ort-192.webp\",\"contentUrl\":\"https:\\\/\\\/i10x.ai\\\/news\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/christopher-ort-192.webp\",\"caption\":\"Christopher Ort\"},\"description\":\"Christopher Ort handles SEO, AI agent workflows and PR & partnerships at i10x.ai, the all-in-one AI workspace built by i10X Global Pte. Ltd. in Singapore. He is a graduate of Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU). He is the author of \u201cThe Style Penalty\u201d, a study published on SSRN that analyses 1,576 evaluation decisions by four frontier AI models to show how AI-written CVs fare in automated hiring screens; the research has been covered by heise online and t3n.\",\"url\":\"https:\\\/\\\/i10x.ai\\\/news\\\/author\\\/christopher-ort\",\"sameAs\":[\"https:\\\/\\\/i10x.ai\\\/blog\\\/author\\\/christopher-ort\"],\"worksFor\":{\"@id\":\"https:\\\/\\\/i10x.ai\\\/news\\\/#organization\"},\"alumniOf\":{\"@type\":\"EducationalOrganization\",\"name\":\"Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU)\",\"url\":\"https:\\\/\\\/www.fau.eu\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Tampered ML Benchmark: AI Misses Research Flaws - i10X News","description":"The Tampered ML benchmark tests whether AI models spot planted flaws in 1,858 ML experiments. Claude Opus 5 leads at 77.9%, but order bias trips up models.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research","og_locale":"en_US","og_type":"article","og_title":"Tampered ML Benchmark: AI Misses Research Flaws - i10X News","og_description":"The Tampered ML benchmark tests whether AI models spot planted flaws in 1,858 ML experiments. Claude Opus 5 leads at 77.9%, but order bias trips up models.","og_url":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research","og_site_name":"i10X News","article_published_time":"2026-10-26T08:12:26+00:00","og_image":[{"width":1344,"height":768,"url":"https:\/\/news.i10x.ai\/wp-content\/uploads\/2026\/10\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp","type":"image\/webp"}],"author":"Christopher Ort","twitter_card":"summary_large_image","twitter_creator":"@i10x_ai","twitter_site":"@i10x_ai","twitter_misc":{"Written by":"Christopher Ort","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"NewsArticle","@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research#article","isPartOf":{"@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research"},"author":{"name":"Christopher Ort","@id":"https:\/\/news.i10x.ai\/#\/schema\/person\/d410bb7ee72febefca2c9c369939219a"},"headline":"Tampered ML: Even the Best AI Models Miss One in Five Planted Research Flaws","datePublished":"2026-10-26T08:12:26+00:00","mainEntityOfPage":{"@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research"},"wordCount":1258,"publisher":{"@id":"https:\/\/news.i10x.ai\/#organization"},"image":{"@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research#primaryimage"},"thumbnailUrl":"https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp","articleSection":["Benchmarks &amp; Evals","Studies"],"inLanguage":"en-US","dateModified":"2026-10-26T08:12:26+00:00"},{"@type":"WebPage","@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research","url":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research","name":"Tampered ML Benchmark: AI Misses Research Flaws - i10X News","isPartOf":{"@id":"https:\/\/news.i10x.ai\/#website"},"primaryImageOfPage":{"@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research#primaryimage"},"image":{"@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research#primaryimage"},"thumbnailUrl":"https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp","datePublished":"2026-10-26T08:12:26+00:00","description":"The Tampered ML benchmark tests whether AI models spot planted flaws in 1,858 ML experiments. Claude Opus 5 leads at 77.9%, but order bias trips up models.","breadcrumb":{"@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research#primaryimage","url":"https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp","contentUrl":"https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/tampered-ml-benchmark-llm-judgment-ml-research-title.webp","width":1344,"height":768,"caption":"Illustration: i10X"},{"@type":"BreadcrumbList","@id":"https:\/\/news.i10x.ai\/tampered-ml-benchmark-llm-judgment-ml-research#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/news.i10x.ai\/"},{"@type":"ListItem","position":2,"name":"Tampered ML: Even the Best AI Models Miss One in Five Planted Research Flaws"}]},{"@type":"WebSite","@id":"https:\/\/news.i10x.ai\/#website","url":"https:\/\/news.i10x.ai\/","name":"i10X News","description":"Stay updated with the latest news and announcements from i10X.","publisher":{"@id":"https:\/\/news.i10x.ai\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/news.i10x.ai\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/news.i10x.ai\/#organization","name":"i10X","url":"https:\/\/news.i10x.ai\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/news.i10x.ai\/#\/schema\/logo\/image\/","url":"https:\/\/i10xnews.kinsta.cloud\/wp-content\/uploads\/2026\/10\/i10x-logo-180.png","contentUrl":"https:\/\/i10xnews.kinsta.cloud\/wp-content\/uploads\/2026\/10\/i10x-logo-180.png","width":180,"height":180,"caption":"i10X"},"image":{"@id":"https:\/\/news.i10x.ai\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/i10x_ai","https:\/\/www.linkedin.com\/company\/i10x-ai","https:\/\/instagram.com\/i10x.ai","https:\/\/www.youtube.com\/@i10X_ai_official"]},{"@type":"Person","@id":"https:\/\/news.i10x.ai\/#\/schema\/person\/d410bb7ee72febefca2c9c369939219a","name":"Christopher Ort","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/christopher-ort-192.webp","url":"https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/christopher-ort-192.webp","contentUrl":"https:\/\/i10x.ai\/news\/wp-content\/uploads\/2026\/10\/christopher-ort-192.webp","caption":"Christopher Ort"},"description":"Christopher Ort handles SEO, AI agent workflows and PR & partnerships at i10x.ai, the all-in-one AI workspace built by i10X Global Pte. Ltd. in Singapore. He is a graduate of Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU). He is the author of \u201cThe Style Penalty\u201d, a study published on SSRN that analyses 1,576 evaluation decisions by four frontier AI models to show how AI-written CVs fare in automated hiring screens; the research has been covered by heise online and t3n.","url":"https:\/\/i10x.ai\/news\/author\/christopher-ort","sameAs":["https:\/\/i10x.ai\/blog\/author\/christopher-ort"],"worksFor":{"@id":"https:\/\/i10x.ai\/news\/#organization"},"alumniOf":{"@type":"EducationalOrganization","name":"Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU)","url":"https:\/\/www.fau.eu"}}]}},"_links":{"self":[{"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/posts\/6607","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/comments?post=6607"}],"version-history":[{"count":1,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/posts\/6607\/revisions"}],"predecessor-version":[{"id":6610,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/posts\/6607\/revisions\/6610"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/media\/6605"}],"wp:attachment":[{"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/media?parent=6607"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/categories?post=6607"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/tags?post=6607"},{"taxonomy":"company","embeddable":true,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/companies?post=6607"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/i10x.ai\/news\/wp-json\/wp\/v2\/products?post=6607"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}