Quick Take
- The finding: On Chatbot Arena, verdicts on the very same unchanged AI models got harsher every month. Within fixed pairs, “both answers are bad” votes rose by 1.71 percentage points a month on a mean of 17.1%, across 1,260,967 blind votes from May 2023 to August 2024.
- Why it is convincing: The rise shows up within the same voter, on the same pair, for newcomers from their first vote, in every major language and for pinned OpenAI and Anthropic snapshots. Which model wins, however, did not change.
- What it means: “The model got dumber” can be true, but it can also be users raising the bar. Absolute ratings of AI need a fixed reference; head-to-head rankings held steady.
Complaints that ChatGPT, Claude or Gemini “got worse” follow almost every popular model. Sometimes the model behind a name really changes. A working paper by Zihan Chen and Di Zhu (Stevens Institute of Technology) and Lei Nico Zheng (University of Massachusetts Boston), “AI Isn’t Getting Dumber, We’re Getting Pickier: Rising Standards in Human Evaluation of Large Language Models,” tests the other explanation: that the judges change. Their lab is Chatbot Arena’s public vote log, where anonymous models compete blind and many of them are frozen versions that never change.
What the study shows
- Same models, harsher verdicts: For the same pair of frozen models, the share of “both are bad” votes rose by 1.71 points a month (SE 0.18), and by 1.91 points a month in 2024, when 92% of the votes were cast. By August 2024 it stood 17.8 points above its May 2023 level.
- No regression needed: In all fourteen pairs judged at least 50 times in both mid-2023 and 2024, the share rose, from 15.5% to 23.3% on a weighted basis. That includes gpt-4-0613 against claude-2.0.
- The bar moves, not the ranking: The extra rejections came out of decisive votes; ties stayed flat. In 2024 the stronger model’s share of decisive votes showed no trend (0.00 points a month, SE 0.13).
- People get pickier: Within the same voter identifier, rejections rose 1.90 points a month; for the same voter on the same pair, 2.30. Newcomers arriving later were harsher from their first recorded vote (+1.25 points per month of arrival).
- No momentary contrast: Seeing a strong model in the previous battle did not make the next verdict harsher (−0.11 points per 100 rating points, not significant).
- A treadmill: The average model in battles gained 245 Elo points, so an August 2024 model would be expected to score about 78% against a May 2023 one. Yet platform-wide “both are bad” votes were 19.4% in May 2023 and 20.6% in August 2024.
| Outcome (frozen pairs) | Change per month | Mean |
|---|---|---|
| “Both are bad” | +1.71 pp | 17.1% |
| One answer wins | −1.62 pp | 63.9% |
| Tie | −0.09 pp | 19.0% |
| Stronger model wins (decisive votes) | +0.13 pp | 66.7% |
| Stronger model wins, 2024 only | 0.00 pp | 66.5% |
Why it matters: a natural experiment in blind judging
Arena happens to offer what the question needs: unchanged products, judged blind by the public, over more than a year, with every verdict time-stamped. The authors hold the model pair fixed with pair fixed effects, so any trend must come from the judges, the prompts or the platform, and then test each alternative. Prompt controls (language, code, math, seven difficulty tags, conversation length, refusals) barely move the estimate, from +1.71 to +1.57. Under Oster’s coefficient-stability test, hidden prompt differences would need to be 36 times as strong as the observed ones to erase the trend. Interface changes were tracked via 56 commits to the battle page; jumps at major changes looked like jumps at placebo dates. Across 28 robustness specifications the trend is positive in every one, ranging from +0.97 to +2.61 points a month. The final test is out of sample: month effects estimated from other models’ pairs cut the average per-model trend from +1.40 to +0.19 points a month.
Method in brief
- Data: LMSYS’s public Arena battle log released in August 2024 (1,799,991 battles). From May 1, 2023, when the “both are bad” button existed, 1,795,852 blind battles; 1,260,967 of them pair two frozen models.
- Frozen models: 33 dated API snapshots (such as gpt-4-0314 or claude-3-opus-20240229) and 73 fixed open-weight models; 23 aliases that may change were excluded. 2,231 frozen pairs.
- Analysis: Linear probability models with pair fixed effects and standard errors clustered by pair and week; judge identifier and identifier-by-pair fixed effects; arrival cohorts using first votes; Bradley–Terry ratings to define the stronger model.
- Serving check: Answer-length trends identified 8 length-stable models out of 40 checked, used as a separate test against hidden serving changes.
What remains open
- Old data: The log ends in August 2024 and covers models like GPT-4 and Claude 3. Whether standards keep rising at this pace with today’s models is untested.
- Who votes: Arena voters are a self-selected, AI-curious public. The judge identifier tracks a device or network, not a person.
- No text: The log has no prompt or answer text, so identical prompts cannot be compared directly.
- Mechanism: The data cannot separate a higher threshold from closer scrutiny, nor link verdicts to each voter’s exposure to other AI. Platform quality rose almost in step with calendar time (correlation .99).
- Status: A working paper, not peer reviewed; no code release or funding statement is given, though the underlying data are public.
Context: from “is ChatGPT changing?” to “are we?”
Chen, Zaharia and Zou showed that the March and June 2023 versions of the model behind ChatGPT behaved differently on several tasks, so model drift is real. Work on Arena itself found rankings sensitive to sampling (“The Leaderboard Illusion”) and to dropping a handful of votes. This paper adds a third factor, drifting human standards, and borrows theory from marketing and psychology: adaptation-level theory and the “hedonic treadmill”. It mirrors reputation inflation on online marketplaces, but in reverse. It also gives context to recent user complaints we analysed in the data behind Claude Opus 5.5 “nerf” claims and to our coverage of bias in LLM judges.
What it means for AI teams and buyers
- Read complaints carefully: A falling satisfaction score does not prove a regression. Keep frozen reference models or fixed prompt-answer pairs in your evaluations to tell product changes from rising expectations.
- Prefer relative judgments over time: Head-to-head preferences stayed stable; absolute “good enough” labels drifted by roughly 21 points a year.
- Time-stamp human feedback: Acceptability labels collected in different periods encode different standards, which matters for RLHF datasets and quality thresholds.
- Standing still loses: For vendors, a model that does not improve will be judged as declining as the market moves.
More on our Benchmarks & Evals page.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of Chen, Zheng and Zhu (2026), “AI Isn’t Getting Dumber, We’re Getting Pickier: Rising Standards in Human Evaluation of Large Language Models,” arXiv:2611.14333v1 (cs.HC, working paper, submitted November 10, 2026), including the appendix tables. All numbers come from the paper. The chart was redrawn by i10X from Tables 2–4; the paper is licensed CC BY 4.0. No funding or conflict-of-interest statement is included. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
This is the most useful reality check on “AI got worse” threads we have seen: it does not deny model changes, it shows that the yardstick moves too. The lesson for anyone running evals is simple and cheap: keep a few frozen references in the mix. The lesson for users is humbling: yesterday’s wow answer is today’s “both are bad”.
