Two studies of AI tutors were posted to arXiv in the last three weeks. The first, from Shihao Yang, Marshall Van Alstyne, and Chrysanthos Dellarocas, went up on September 21 and tested which parts of a tutor’s design changed how much students learned. The second, from Daniel Domínguez Figaredo and Rafael Fernández De la Cruz, went up on October 8. They trained a tutor to score higher on a measure of teaching quality and then asked educators to judge what it produced.
Together they bear on what an AI tutor should ask students to do and how a school would know it’s working.
Structure beat open access
The Yang study ran in a graduate corporate finance course in an online MBA program. Eighty-six students who volunteered were randomly split into two groups. One group got a tutor built around the course materials. It opened each session with a few recall questions on earlier material, and it asked students to attempt a problem before it explained anything. The other group got nothing from the researchers, and nothing stopped them from using ChatGPT, Gemini, Claude, or any other tool, which many of them did. That second condition is close to where most of our students are right now.
After five weeks, the students with the structured tutor gained 6.63 more points on a 55-point test than comparable students in the other group, under the analysis the researchers registered before the study began. Most of the difference showed up in written answers. The researchers scored short answers on a scale that separates listing ideas from connecting them. The share of answers that connected ideas rose from 8 percent to 49 percent with the tutor and from 8 percent to 27 percent without it. The instructor’s own final exam, which the researchers didn’t write, pointed in the same direction by 2.6 points out of 100.
The authors are open about the limits. Only 46 students finished the post-test and passed the quality checks, so the main estimate rests on 29 students in one group and 17 in the other. A simpler version of the same analysis gives a smaller gain of 3.56 points that misses statistical significance. A panel of AI models scored the short answers without knowing which group each student was in, and a human scoring pass is still to come. These were adults in one quantitative course, and my students are eleventh graders writing history essays. I’d read the result as a signal that needs a high school replication.
Voice cost more and taught the same
Inside the tutor group, each student switched every week between talking to the tutor and typing to it. In voice weeks, students went back and forth with the tutor nearly twice as much, and their stated preference moved toward voice as the weeks went on. Their weekly mastery scores were statistically the same in voice weeks and text weeks, and voice cost 2.8 times as much to run.
The paper’s last line reads, “Making an AI more humanlike does not by itself make it more educational.” Much of what’s sold to schools is pitched on how natural a tutor feels to talk to. Students will like those features, and in this study they came to prefer voice while learning the same amount either way. It also shows how an engagement number can mislead. If a school measured success by how much students interacted with the tutor, voice would have looked like the better product.
A better score from a worse tutor
The second study worked on a different problem. The researchers started with Google’s Gemini 3.1 Pro Preview acting as a tutor. For each simulated learner, it planned a sequence of five learning activities. The researchers built a metric that scored each activity choice against that learner’s situation, which they describe as the standard approach for automated scoring of teaching. They ran the tutor through 2,000 learner scenarios, took the weakest cases, and fine-tuned an open-weights model, Qwen3-32B, to do better on the metric.
On the corrected cases, the metric rose from +0.05 to +0.42. Then 31 trained educators rated the plans without knowing which version they were reading, on a scale of 1 to 5. Their average rating fell from 4.46 to 3.03, while plans the researchers didn’t touch kept the same ratings.
The metric scored each of the five activities on its own and averaged the scores, so the highest possible score came from finding the single best activity and repeating it five times. The fine-tuned model learned to do that, in every case, including 200 cases it had never seen. The raters noticed: their written comments named the repetition, and they gave lower ratings to the plans they called monotonous. The metric had no way to see it, because it treated the learner at the fifth activity as the same person as the learner at the first. The paper’s title calls this “the learner who does not learn.”
This study has limits too: the raters were graduate students rather than practicing teachers, and the learners were written scenarios. The fine-tuned model also copied stock descriptions from its training material, 748 of 750 of them word for word, and some of the drop in ratings may come from that. The math explains why the metric rewarded repetition either way, but the paper can’t fully separate how much of the rating drop came from the repetition and how much from the copied text.
What both studies mean for a school
In the Yang study, the design of the tutor made the difference, and the feature students liked most added cost without adding learning. In the Domínguez Figaredo study, a number built to capture good teaching rewarded a tutor that assigned the same activity five times in a row.
The Yang results push on something I argued last month in The Gap Between AI Deployment and Student Literacy. There I put most of the weight on schools rolling out AI faster than they teach students to use it. Both groups in this study had the same course and the same instructor. The difference between the groups was the tool. A student with an open chatbot could take an answer whenever they wanted one, while a student with the structured tutor was asked to recall, attempt, and explain. I still think training lags deployment, but I’d now give the design of the tool more of that weight.
The second study bears on how schools will be shown that a tutor works. Schools will get dashboards with mastery percentages, minutes of engagement, and alignment scores. Each of those numbers measures something, and a tool can be built or tuned to push any of them up. The 31 raters caught what the metric missed, because they read each plan as a sequence a student would have to sit through.
Takeaway for Teachers
If your school uses or is piloting an AI tutor, ask for five full session transcripts from your own students this week and read them start to finish. Check three things: whether the tutor asks students to try before it explains, whether it asks them to explain their reasoning in writing, and whether the activities change as the student gets things right or keep repeating the same move. Then take two questions to whoever chose the tool: what does it ask students to produce, and what number does the company use to say it works? If your school hasn’t adopted a tutor, take one homework question from this week, put it into the chatbot your students use, and see whether it teaches or hands over the answer.
If you’re thinking through questions like this one, about what an AI tutor should ask of your students and how you’d know it’s working, my book goes deeper. The AI Doesn’t Know Your Students is available on Amazon and at shouldiuse.ai/book.
David Jacobson is a high school history teacher. He writes about AI, education, and the messy intersection of the two at shouldiuse.ai.
