Quick Take
- The finding: In a preregistered experiment with 1,586 Americans, warnings on a politically biased chatbot reduced its influence only for some users and some tasks. No disclaimer worked across both parties.
- The split: For Democrats, explaining why the model may be biased cut opinion shifts on conservative-leaning topics. For Republicans, a specific political-bias warning cut how far their budget decisions followed the model.
- The catch: One week later, none of the disclaimer effects was still statistically detectable. Lower trust in the AI also did not reliably mean less persuasion.
AI chatbots usually carry a small warning that they “can make mistakes”. A new preprint from researchers at Stanford University and the University of Washington tests whether better-worded warnings can protect users when the model is actually politically skewed. In “Whose Guardrail? User Identity Shapes the Effectiveness of Model Disclaimers in Mitigating LLM Political Persuasion”, Jillian Fisher, Jennifer Pan, Yulia Tsvetkov, Katharina Reinecke and colleagues find that it depends heavily on who is reading the warning, and on what they are deciding.
What the study shows
Every participant talked to a version of GPT-4o-mini that was secretly biased against their own party. Democrats got a conservative-leaning model, Republicans a liberal-leaning one. Before chatting, each person saw one of four disclaimers in a yellow box.
| Condition | Disclaimer text shown to participants |
|---|---|
| Generic (control) | AI models are imperfect and can make mistakes. They may give incorrect answers. |
| Specific risk | AI models are unbalanced and can be politically skewed. They may give politically one-sided responses. |
| Generic + cause | This AI was trained on imperfect internet data that can contain faulty information. Therefore, it can produce incorrect responses. |
| Specific + cause | This AI was trained on unbalanced internet data that can contain politically skewed information. Therefore, it can produce politically one-sided responses. |
- Democrats: A causal explanation of the bias reduced opinion shifts toward the model on conservative-leaning topics (coefficient −0.36 on a −3 to 3 scale, p<.05). The specific political-bias warning had a smaller, only marginal effect (−0.26, p<.10). Neither helped in the budget task.
- Republicans: The specific political-bias warning reduced how far budget allocations moved toward the model’s preferred programs by 6.32 percentage points (p<.05). Explaining the cause did not have a significant effect, and opinion shifts on topics did not change.
- Trust is a poor proxy: Republicans reported higher trust in the AI than Democrats in every condition. Participants who correctly recalled their disclaimer reported lower trust in both parties, but only Democrats showed a matching, marginal drop in persuasion.
- Bias often went unnoticed: 63.95% of Democrats but only 48.36% of Republicans said they noticed the model’s bias.
- Short-lived: In the one-week follow-up (1,242 participants), no disclaimer effect was statistically significant. Several coefficients kept their direction but shrank.
The exploratory analyses add a twist on “AI literacy”. People who rated their own AI knowledge higher were persuaded less in all three tasks. People who use AI more often were persuaded more, and AI use was the strongest predictor of trust. Formal AI education raised trust but did not predict persuasion.
Why it matters: A warning label is not one-size-fits-all
Disclaimers are the cheapest safety measure in AI products, and most companies use a single generic line for everyone. This study shows why that may be too simple. The same text helped one group in one task and did nothing for another group in another task. The authors suggest one reason may be that people judge the credibility of the warning itself differently, similar to partisan differences in trust toward fact-check labels. They note that they did not measure this directly.
The design is solid for a disclaimer study. It was preregistered, used a 2×2 experiment with party as a blocking factor, compared each warning against a realistic generic disclaimer rather than no warning at all, and checked whether effects lasted a week. The authors also report where they deviated from the preregistration.
Method in brief
- Sample: 1,600 US adults recruited via Prolific (800 Democrats, 800 Republicans); 14 who failed bot checks were excluded, leaving 1,586. Participants were paid $15 per hour.
- Tasks: A topic opinion task on lesser-known issues (covenant marriage, unilateralism, postal banking, nonprofit tax uniformity) with 3 to 20 chat turns, and a budget task in which participants acted as a mayor splitting funds across public safety, education, veteran services and welfare.
- Bias: The model was steered with a bias-inducing prefix added to each user message on the backend, the same method as an earlier study by the lead author. Participants were debriefed about the bias after the main study.
- Analysis: OLS regressions with robust standard errors, run separately per party and one-tailed for the preregistered hypotheses. Personality and AI-familiarity analyses were exploratory and Holm-corrected.
What remains open
- One model, one country: Only GPT-4o-mini with one prompting method was tested, and only US partisans. The authors say results need replication in other political contexts.
- Online sample: Prolific participants are not representative of the US population.
- Marginal and partial effects: Several effects are significant only at one-tailed p<.10, and the hypotheses were supported only in part.
- Follow-up is hard to read: The smaller follow-up sample has less statistical power, and the debriefing about the bias happened before the follow-up survey, which may have changed answers.
- Correlational extras: The AI-literacy and personality findings are hypothesis-generating, not causal. The proposed fixes, such as standardized disclosures or third-party audits, were not tested.
- Funding: The work was supported by NSF and DARPA grants and gift funding from Coefficient Giving. The paper has no separate conflict-of-interest statement. It is a preprint without peer review.
Context: Warnings, literacy and persuasive chatbots
The result complements an earlier study covered on i10X, in which a short AI-literacy warning roughly halved AI political persuasion. The new paper adds that the effect of a warning can depend on the user’s party and on the decision at hand. It also matters for companies like OpenAI, whose model was used here, and for regulators who see labels as a key transparency tool. More research on these questions is collected under Society & Economy and Safety, Ethics & Society.
What it means for AI product teams
- Be specific: A concrete warning about political one-sidedness, or an explanation of where bias comes from, beat the generic “can make mistakes” line in some settings.
- Test with different user groups: Check whether a disclaimer works across user groups and tasks before rolling it out.
- Measure behavior, not just trust: A drop in self-reported trust did not reliably mean people followed the model less.
- Don’t rely on labels alone: The authors point to deliberate friction, such as an “I understand” click or a checklist before accepting AI advice, and to training that builds evaluation skills rather than just familiarity.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated illustration. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of the preprint (arXiv:2611.05347v1, submitted November 3, 2026), including the appendix. All numbers come from the paper; i10X did not rerun the experiment. The chart was drawn by i10X from Table 8, because the paper is licensed CC BY-NC-SA 4.0. AI tools assisted with research and drafting; the selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
Right now, the standard disclaimer is a legal footnote. This study suggests it could be designed as a real intervention: specific, tested on different users, and backed by other safeguards. For chatbots that millions use to form opinions, “AI can make mistakes” is probably not enough.
