A research team in the UK took 761 psychology assignments that university staff had already marked and asked three AI models to mark them again. The study, posted to arXiv on October 13 by Giulio Corsi and six colleagues from Cambridge, Manchester Metropolitan, Nottingham, and Lancashire, used GPT-5.4, Claude Opus 4.6, and Gemini 3 Flash. The team tried 27 prompt designs and four ways of combining the models' marks, so the tools had a fair chance to do well.
All three models pulled marks toward the middle. On average, work that human markers had placed below 50 got 11.6 more points from the machines, and work that humans had marked as a First, the top UK degree band, got 11.9 fewer. Assignments in the middle of the range came out close to where the humans put them.
Where the error lands
The size of the miss depends on where an assignment sits. Marks in the 50s were over-marked by about three points, and marks in the high 60s were under-marked by about four. The errors at the ends of the range were three to four times larger. A student whose work sits below 50 is close to failing and needs to know it, and the students doing First-class work were the ones the machines marked down the most.
The authors tried the obvious fixes. Combining the three models didn't help, because all three made their mistakes in the same direction, so averaging them kept the error. Statistical corrections fitted on one set of essays and carried over to another removed only part of the problem. For every model, where an assignment sat on the human scale explained 62 to 69% of the variation in the machine's error. The team also measured 33 features of each piece of writing, including vocabulary range, readability, sentence complexity, error rate, and word count, and the machine marks tracked those surface features more strongly than the human marks did. The authors connect this to older research on automated essay scoring, which found systems that could match human scores while rewarding length and sophisticated vocabulary instead of the skills the assignment was meant to measure.
It depends on where you test it
Agreement also changed from one university to the next. The best setup reached an agreement score, called kappa, of 0.616 at Cambridge, 0.420 at Nottingham, and 0.345 at Manchester Met. Kappa corrects for chance agreement, so the Manchester Met figure is much closer to chance than the Cambridge one. GPT was the strongest model at Cambridge and among the weakest at Manchester Met, and no model or prompt came out on top at all three universities.
The authors describe agreement as "a joint product of model, prompt and institution," and they conclude that testing a tool in one setting overstates how ready it is for any other.
A department can run a small version of the researchers' check before it trusts any grading tool: take a set of essays your teachers have already graded, run them through the tool, and compare the two sets of grades, paying the most attention to the top and bottom of the range.
Human markers aren't perfectly consistent either, and the paper says so. In published studies of UK psychology marking, correlations between first and second markers have ranged from 0.47 to 0.93. The trouble with the models was that all three erred in the same direction, so adding more of them gave no second opinion.
AI detectors on Kazakh
Two days earlier, on October 11, Mansur Ozaman posted a study of AI-text detectors tested on Kazakh. He assembled 3,811 human-written Kazakh passages, all written before ChatGPT existed, and ran five English-trained detectors from the Hugging Face Hub on them. The detectors flagged up to 81.4% of the human passages as machine-written, along with 79.6% of a set of published student essays. When Ozaman adjusted the detectors so they stopped accusing human writers, they caught at most 1.3% of the machine text.
These are open-source detectors, not the commercial products most schools license, and Kazakh is a long way from English. Ozaman's own detector, built for Kazakh, did far better, flagging 0.9% of human passages, though he notes that its settings were chosen with the test results in view.
What this means for my students
I teach AP history at an international school in Shanghai, and many of my students write in a second or third language. A lot of them write careful, plain English, and some of the sharpest arguments I read arrive in short, simple sentences.
The marking study found that machine marks lean harder on vocabulary range and sentence complexity than human marks do. It didn't test students like mine, so I can't say exactly how a tool would treat them. A tool that gives extra weight to polish could grade a sharp argument in plain English as average. Those are the essays where a wrong grade costs the most, because the feedback those students need differs from what the machine would give.
What the studies can't show
The marking study used university psychology assignments on the UK percentage scale. The prompts ranged from no rubric at all to each university's full marking criteria, but the models never saw an example essay marked by the department. A tool given worked examples from your own grading might do better, and the authors didn't test high school writing at all. The detector study covers one language and a handful of open tools.
Takeaway for Teachers
If you use AI to grade or pre-grade anything this week, grade the essays it ranks highest and lowest by hand, since the Corsi study found the models furthest off on the strongest and weakest work. Then pull a few from the middle of the pile as well, including any essay whose machine grade surprises you given what you know about the student, because a tool that squeezes grades toward the middle can bury a strong essay there. If you have time for one more check, grade ten essays on your own before you run the tool, then compare your grades with its grades to see how it does on your students and your assignment. If a detector flags a student who writes in a second language, treat the flag as a reason to sit down with the student and their drafts.
If you're thinking through questions like this one, about which students an AI tool misjudges and why, my book goes deeper. The AI Doesn't Know Your Students is available on Amazon and at shouldiuse.ai/book.
David Jacobson is a high school history teacher. He writes about AI, education, and the messy intersection of the two at shouldiuse.ai.
