Quick Take
- The finding: When AI models screen job applications, the AI model that wrote the cover letter shifts the score, independent of how strong the candidate actually is.
- The size of the effect: In the worst pairing, about 1.8 of 25 strong candidates per job (net +7.3 percentage points) lose their shortlist spot to weaker candidates purely because of which model wrote each letter.
- Who loses: Letters from small, locally run models (Llama-3.1-8B, DeepSeek-R1-8B) win the fewest head-to-head comparisons, so applicants without paid AI access are at a systematic disadvantage.
Job seekers increasingly write cover letters with chatbots, and employers increasingly let language models screen them. A new preprint from the University of Zurich and ETH Zurich, “AI evaluators favor job applications written by certain AI models, independent of candidate quality” by Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu and Anikó Hannák, tests what happens when both sides of hiring are automated. The answer: the choice of writing model becomes a hidden factor in who gets shortlisted.
What the study shows
The authors had 11 models write 5,500 cover letters for 500 real, anonymized CVs across ten job ads, then let nine API models score the applications 414,000 times on a 1 to 10 scale. Half of the CVs per job were a strong match for the job ad (high-fit), half a weaker match (moderate-fit).
- Evaluators agree on candidates, not on letters: On CVs alone, evaluators rank candidates almost identically (average Spearman ρ 0.92) and separate the two tiers clearly (2.25 points). But when they rank the 11 letters written for the same candidate, agreement is only ρ 0.63 with the letter alone and 0.27 with CV and letter together.
- Some writers are favored: With the cover letter alone, letters from Gemini 3.5 Flash scored on average 0.45 points above the evaluator’s mean for the same candidate, followed by GPT-4o mini (+0.44) and Gemini 3 Flash (+0.36), while Llama-3.1-8B letters fared worst (−0.53).
- Some evaluators are more biased: DeepSeek V3.2 became the most biased evaluator once the CV was added, with a cumulative bias score that rose from 10.7 to 14.9. On average it ranked GPT-5 letters 2.0 places above the neutral midpoint among the 11 letters and Llama letters 3.1 places below. Gemini 3.5 Flash was the most neutral in that setting (6.3).
- Self-preference exists but varies: In the combined condition, DeepSeek V4 Flash (+0.22), GPT-5 mini (+0.19) and Gemini 3.5 Flash (+0.14) favored their own letters, while Claude Haiku 4.5 showed essentially none (−0.001).
- Real rank reversals: When strong candidates use Llama-3.1-8B and weaker ones use DeepSeek V3.2, the net displacement rate is +7.3% ± 3.2%, on top of a same-writer noise floor of 8.2% on average. Single evaluators (GPT-4o mini, DeepSeek V3.2) reach up to +12.8%, about 3.2 of 25 candidates.
| Measure (CV + cover letter) | Result |
|---|---|
| Top writer models by average win rate | Gemini 3.5 Flash 56%, GPT-5 mini 56%, DeepSeek V3.2 54% |
| Weakest writers | Claude Haiku 4.5 47% (weakest API model), DeepSeek-R1-8B 42%, Llama-3.1-8B 36% |
| Most biased evaluator | DeepSeek V3.2, cumulative bias 14.9 |
| Most neutral evaluator | Gemini 3.5 Flash, cumulative bias 6.3 |
| Worst-case net displacement | +7.3% ± 3.2% (≈1.8 of 25 high-fit candidates per job) |
Why it matters: AI cover letter bias is a hidden hiring factor
The study separates writer effects from candidate quality cleanly. Every CV gets a letter from every model, so differences between letters for the same candidate can only come from the writing model. Scores are reproducible (standard deviation at most about 0.5 points across four independent runs), the evaluators agree with the study’s own quality split on 86 to 94% of candidates, and the displacement estimate is already corrected for ordinary scoring noise.
The authors also explain part of the bias. A regression on writing-style features shows that semantic closeness to the job ad is the strongest driver of evaluator preference, with a coefficient about three times that of readability. GPT-4o mini letters, for example, lean strongly toward the wording of the job ad. Text features explain the preference about as well as knowing the writer’s identity (0.910 vs. 0.911), which suggests the effect runs through style rather than hidden watermarks.
Method in brief
- Data: Public CV and job-ad datasets; ten jobs with 50 CVs each, split into 25 high-fit and 25 moderate-fit by embedding similarity to the job ad.
- Writers: GPT-4o mini, GPT-5 mini, GPT-5, Claude Haiku 4.5, Claude Sonnet 4.6, Gemini 3 Flash, Gemini 3.5 Flash, DeepSeek V3.2 and DeepSeek V4 Flash via API, plus Llama-3.1-8B and DeepSeek-R1-8B run locally. Prompt: a persuasive 350 to 400 word letter without fabricated credentials.
- Evaluators: The nine API models, each scoring under three conditions (CV only, letter only, CV + letter), four runs per item, with a fixed rubric and a gender-neutral sign-off.
- Analysis: Rank-based preference gaps, self-preference, head-to-head win rates excluding self-evaluations, noise-corrected rank displacement and a Ridge regression on style features.
What remains open
- No human recruiters: The study measures AI evaluators only; it does not test whether human recruiters react the same way.
- Artificial pairings: CVs and job ads were matched by the researchers, which keeps absolute scores low (around 4 out of 10 for high-fit candidates on CV alone).
- Model churn: The ranking of writer models will shift with each new release, and the authors say so. The mechanism is the durable finding, not the leaderboard.
- Size and hosting are confounded: Both local models are about 8B parameters, so the study cannot separate model size from access cost.
- Limited openness: Code and the generated letters and scores are available only on request, not as a public release. The paper’s overview figure quotes “~1.4/25” displaced candidates, while the text and appendix give 1.8; we use the text value.
Context: AI on both sides of the hiring table
Earlier work found that LLM evaluators prefer resumes written by themselves and that easy AI writing makes cover letters a weaker signal for employers. This study adds a cross-model view: the bias does not only favor one’s own outputs but specific providers, such as Google‘s Gemini Flash models and OpenAI‘s GPT-5 mini. It fits our earlier coverage of racial bias in LLM resume screening and the AI-to-AI recruiting loop. More research of this kind is collected in Society & Economy.
What it means for employers and applicants
- Employers: Do not let a single LLM screener make shortlist decisions on cover letters. Score CVs separately, rotate or ensemble evaluator models, and audit how often the screener reverses CV-based rankings.
- Vendors of hiring tools: The EU AI Act treats AI used in recruitment as high-risk. Writer-model bias is a measurable fairness risk that audits can test with exactly this design.
- Applicants: The model you use can matter. Letters that track the wording of the job ad and stay readable scored better, which you can do with any tool.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint (arXiv:2610.41866v1, submitted October 30, 2026), including the appendices. All numbers come from the paper; i10X did not rerun the experiments. The chart was drawn by i10X from numbers stated in Section 2.4; the paper is licensed CC BY 4.0. The work was funded by the Swiss National Science Foundation through NCCR Automation; the authors declare no competing interests. AI tools assisted with research and drafting; the selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
Hiring is turning into one model writing for another. This study shows that the conversation between those models is not neutral: style choices baked into one provider’s model can tip a shortlist. For a decision this consequential, a single model as gatekeeper is not good enough.
