A Stanford study published in March did something methodologically simple: it submitted 600 eighth-grade essays to AI writing coaches, and then resubmitted each one with different descriptions of the student who wrote it. Black. White. Hispanic. Male. Female. High-achieving. Unmotivated. English learner.
The feedback shifted every time. On the same essay.
Essays attributed to Black students got praise and talk of leadership. Essays attributed to Hispanic students or English learners got grammar corrections and prompts about proper English. Essays attributed to white students got pushed on argument quality, evidence, and clarity. Female students got more affectionate language. Students labeled unmotivated got encouragement. Students labeled high-achieving got direct critique.
The word the researchers use for the most consistent finding is feedback withholding. Some students are consistently given less to work with.
More praise. Lower bar.
The selling point of AI writing coaches is personalization. The pitch is: give the tool context about your student and the feedback will be appropriate to where they are. That pitch is the problem. The study is not showing that AI gives bad scores. It is showing that AI reads demographic context and adjusts its expectations accordingly, using stereotypes as a proxy for knowing the student.
Personalization was supposed to solve a real problem. It is producing a familiar one.
There is a dimension to this that most schools have not thought through. Teachers are unlikely to type a student’s race into an AI feedback prompt. But many learning management systems collect detailed student data: race, language status, prior achievement, disability. As AI embeds itself in these platforms, it will have access to context no teacher consciously provided. And even without explicit labels, the Stanford researchers note, AI can sometimes infer aspects of a student’s identity from their writing itself.
The demographic profile does not need to be entered intentionally to shape what comes back.
A fifth grade teacher in Texas who uses her own AI writing coach already intuited this. She strips student identity from every prompt. She describes the assignment goal and her rubric, and she asks for feedback against that. She does not tell the tool who the student is, because she already knows it would color the response.
She is right, and now there is data behind it.
The study’s lead author, Mei Tan, made one other observation: the AI’s default feedback, even when given no student description at all, was not good. She described it as focused on corrections and discouraging. The demographic bias layered on top is its own problem. The baseline is a separate one.
The practical question for any teacher using an AI feedback tool right now is not whether to use it. It is what the platform already knows about your students, whether you put it there or not.
This is one of the central arguments in The AI Doesn’t Know Your Students — if the idea that AI substitutes category for individual knowledge concerns you, the book works through it across every layer of classroom practice.
David Jacobson teaches high school history in Shanghai. He writes about AI and education at shouldiuse.ai and is the author of The AI Doesn’t Know Your Students (print | Kindle).
Source: Tan, Phalen & Demszky, “Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback,” LAK26 (2026). arxiv.org/pdf/2603.12471
Also covered: Hechinger Report (April 27, 2026) · Education Week (June 2026) · KQED MindShift (April 27, 2026)
