Quick Take
- The finding: In a pre-registered randomized trial with 7,200 students in 55 applied STEM classes at 19 universities, AI tools that instructors designed for their own courses raised scores on proctored, AI-free exams by 0.32 standard deviations at the student level and 0.26 at the class level.
- Why it matters: Several recent studies found that generic chatbots or centrally designed AI tutors can make students learn less. This trial suggests the design and ownership of the tool matter more than the technology itself.
- What it means: Universities that want AI to help learning may do better by giving instructors time, training and a safe platform to build their own tools than by rolling out one chatbot for everyone.
In plain English
Many universities worry that AI chatbots let students skip the hard thinking that makes learning stick. This study tested a different approach: instead of handing every class the same chatbot, researchers let each professor design AI helpers that fit their own course, such as practice tutors, coaching bots that ask questions instead of giving answers, and role-play simulations. Students in those classes scored better on supervised paper exams where they could not use AI. They did not, however, attend more or rate their courses higher. For students and parents, the takeaway is that AI in the classroom can help, but how it is designed and who designs it makes the difference.
What the study shows
The debate about AI in higher education has been dominated by warning signs. The authors of this preprint, a consortium of 52 researchers led by Jun Li of the University of Michigan with Robert Bray, Sébastien Martin, Nils Van den Steen and Andrew Wu as shared first authors, cite studies in which students given off-the-shelf chatbots or centrally planned AI tools performed worse. Their paper, posted to arXiv on November 14, 2026, asks whether the result changes when the people who know a class best, its instructors, are the ones who design the AI.
- Learning went up: The student-level effect on proctored exam scores was 0.32 standard deviations (standard error 0.13, P = 0.019, 95% confidence interval 0.06 to 0.59). The class-level estimate was 0.26 standard deviations (standard error 0.12, P = 0.038, interval 0.02 to 0.51). The authors report similar results across three estimators and two inference methods.
- Engagement and satisfaction did not move: Engagement effects were minus 0.07 standard deviations at the student level (P = 0.60) and minus 0.05 at the class level. Satisfaction effects were minus 0.11 (P = 0.29) and minus 0.10 (P = 0.30). None was statistically significant.
- Instructors built very different tools: The interventions spanned 33 types and seven teaching functions. Instructors built course navigation tools (25 classes), individual coaching agents using Socratic questioning (19), tutors for practice and exam preparation (18), interactive simulations (8) and stakeholder role-plays (8). The average class used AI for 3.0 distinct teaching functions.
- Students mostly liked it: In an anonymous survey answered by 2,019 of 4,898 students in treatment classes, 16.1 times as many said the AI aids improved the course as said they worsened it. Practice and exam preparation was the most valued feature (49.2% of 1,414 open answers). 8.6% of those who suggested improvements wanted less AI, and 4.0% raised worries about overreliance.

| Outcome | Student level | Class level |
|---|---|---|
| Learning (proctored exams) | +0.32 SD (P = 0.019) | +0.26 SD (P = 0.038) |
| Engagement | -0.07 SD (P = 0.60) | -0.05 SD (P = 0.60) |
| Satisfaction | -0.11 SD (P = 0.29) | -0.10 SD (P = 0.30) |
For scale, the authors compare the learning gain with a median of 0.10 standard deviations reported for randomized pre-K to 12 interventions and a mean of 0.47 for active learning in undergraduate STEM courses.
Why it matters: a rare large, randomized test
Most evidence on AI and learning comes from a single course, a few hours or weeks of exposure, no random assignment, or tests where students could still use AI. This study addresses each of those gaps. Classes were grouped by subject and level and randomly assigned to introduce their AI tools immediately or one teaching term later, so the control group consisted of comparable courses taught by instructors who had also signed up and received the same training. Outcomes were measured over full academic terms and compared with the same course’s results in earlier years, a difference-in-differences design that accounts for fixed differences between classes and common shocks over time.
The learning measure is the strongest part of the design. Only raw, uncurved scores from AI-proof assessments counted, meaning proctored in-person exams, mostly on paper, or computer exams under lockdown software or in testing centers. Ten class-terms whose exams were not AI-proof were excluded from the learning analysis. The protocol was pre-registered on the Open Science Framework on February 28, 2025, and all 19 institutions gave ethics approval.
Method in brief
- Participants: 19 universities, 42 instructors and 7,200 students in 55 classes (30 undergraduate, 25 graduate), spread over three entry cohorts from spring 2025 to spring 2026. Courses covered statistics, machine learning, optimization, analytics, forecasting and related subjects.
- Intervention: Before randomization each instructor submitted a written proposal that had to go beyond students’ everyday AI use. Instructors in both arms received training video calls every two months, two in-person workshops and access to Faculty Twin, a platform the team co-developed with Google’s Rapid Innovation Team to deploy course agents in a privacy-compliant environment.
- Samples: The learning analysis uses 44 experimental class-terms with 4,768 students, plus earlier terms of the same courses for comparison. Classes with instructor, curriculum or exam changes were excluded under the protocol. The mean consent rate for student-level data was 79.4%.
- Analysis: Ordinary least squares with class and term fixed effects and cluster-robust confidence intervals, checked against two instrumental-variable estimators. The authors also interviewed all 25 instructors who implemented AI.
What remains open
- Only applied STEM: All classes were applied STEM courses. The authors say effects in humanities, social sciences or other settings remain to be tested.
- Persistence is unclear: Few classes were observed in a second or third term with AI, so it is not known whether the gains last.
- Which tool works: The trial lacks the statistical power to compare the different AI designs, so it cannot say whether tutors, simulations or coaching agents drove the effect.
- Some analysis choices came later: The headline regressions are slightly simpler than the pre-registered ones (the authors report nearly identical results for the pre-registered versions), and the rule excluding exams that were not AI-proof was not pre-registered.
- Limited reproducibility: Student data are restricted and available only to qualified researchers under an agreement, and the R analysis code is available to editors and referees on request. The paper is a preprint and has not yet been peer-reviewed.
- Volunteer instructors: Instructors opted in, which is necessary for this design but means the result applies to motivated instructors given substantial support.
Context: a counterpoint to the warning studies
The authors position their result against studies such as Bastani et al. in PNAS (2025), which found that generative AI without guardrails can harm learning in high school mathematics, Fan et al. (2025) on metacognitive laziness, and a 2026 randomized university trial of course-integrated AI tutoring by Liu et al. Hamsa Bastani, the first author of the PNAS study, is a co-author here. The key difference, the authors argue, is bottom-up design: instructors built and refined the tools rather than receiving a single protocol. Readers following the policy side can compare this with our AI in Education Policy Tracker.
What it means for universities and teachers
- University leaders: Fund instructor time, training and a secure deployment platform rather than only licensing a campus-wide chatbot. In this trial, support for instructors was part of the intervention.
- Instructors: The most common designs required students to attempt problems or explain their reasoning before the AI helped. Course-specific practice and exam preparation were what students valued most.
- Assessment: Measure learning with exams where AI cannot be used. Grades on homework or open-book online tests cannot separate learning from AI assistance.
- Edtech buyers: Tools that let instructors customize prompts and course material fit this evidence better than fixed tutors. Our comparison of AI tools for education lists the main options.
More research on AI in teaching is collected on our Education page and in Society & Economy studies.
About the analysis
Researched and written with AI assistance under the editorial responsibility of the i10X News team. The title image is an AI-generated image. Read our AI Use Policy. Spotted an error? Request a correction.
This article is based on the full text of Li, Bray, Martin, Van den Steen, Wu et al. (2026), “Instructor-designed AI improves learning in a randomized trial across 19 universities,” arXiv:2611.19099v1 (cs.AI, submitted November 14, 2026). All numbers come from the paper. The chart was drawn by i10X from numbers in the paper’s Results and Discussion sections; the paper is published under a CC BY 4.0 license. The pre-registration is on the Open Science Framework. Funding came from discretionary research funds of Northwestern University and the University of Michigan; the authors declare no competing interests. AI tools assisted with research and drafting; selection, fact-checking and final editing were done by the i10X editorial team.
i10x Perspective
This is one of the most important pieces of evidence on AI in universities so far, because it measures learning where students could not use AI. The message is not that AI makes students smarter by default. It is that AI helps when teachers shape it around their own course and their own students. That costs time and money, which is exactly what most campus AI rollouts skip.



