This July, in Boston, dozens of students will sit down with school superintendents to draft a model AI policy for American schools. The event is organized by AASA, Day of AI, and MIT RAISE. Students will lead the charge—setting guardrails, writing use policies, protecting privacy, deciding how AI should and shouldn’t show up in classrooms.
It might produce real, useful policy.
But look carefully at what the students are being asked to decide.
The Table Was Already Set
They won’t be deciding whether AI grading tools get purchased. They won’t be asked whether behavior-monitoring software that tracks students around the clock—and, in 29% of cases, generates “risk scores” based on online activity, according to a recent UCSD study—belongs in schools at all. Those decisions were made before the students got to the table. The tools are already deployed, already integrated, already running.
Being invited to write the rules for how you’re managed is not the same as being asked whether this is how you want to be managed. Students understand that distinction. Most policy conversations about student voice in AI skip right past it.
The Lesson Built Into the System
AI systems in schools do a handful of things: they evaluate, they recommend, they flag, they score.
The grading algorithm returns a rubric score on an essay. The personalized learning platform decides you’re ready for the next unit—or not. The behavior-monitoring software notes an unusual search and generates a concern level. The AI tutor suggests the next problem.
None of this is inherently harmful. Some of it is useful. But when every system a student encounters is optimizing them toward a predetermined outcome—a score, a pathway, a clean profile—the cumulative lesson isn’t “think for yourself.” It’s: learn what the system wants, perform for it, get it to say yes. Compliance has been personalized.
Research published this year in Frontiers in Human Dynamics found that AI in education shapes student agency and educational stratification in ways that are uneven and often hidden. Students with more resources are more likely to develop the capacity to interrogate AI tools; students without those resources are more likely to simply follow whatever the system recommends. The technology doesn’t distribute agency—it concentrates it.
Seventy percent of teachers say they worry AI weakens critical thinking and research skills. Ninety-five percent of college faculty fear student overreliance on AI. These concerns are right but slightly off-target. They focus on the cognitive—will students forget how to research?—rather than the structural: are we building systems that train students to seek approval rather than exercise judgment?
A Pattern with Precedents
Industrial-era schools valued punctuality, standardized outputs, and conformity to procedure, because that’s what factories needed. The bells, the rows of desks, the standardized tests—all of it made sense in a world where the premium was on reliable, predictable performance within a fixed system. Schools were optimizing for the labor market their graduates would enter.
We’re in a different moment. The jobs that will carry weight in twenty years are precisely the ones that require people to question systems, propose alternatives, and push back when the algorithm gets it wrong. The entire premium on human work in an AI-saturated economy is the capacity to do what machines can’t: recognize when the rubric is wrong.
The contradiction is hard to miss: we’re deploying AI tools in schools that, structurally, optimize students to be good at what machines are already good at—at the exact historical moment when the premium should be running the other direction.
Two Versions of the Same Classroom
I teach high school history in Shanghai. I watch students navigate AI every day—for homework, for studying, for feedback on writing they’ll never show me. Some of them have developed real judgment about these tools: they know when the AI is wrong, they push back on its suggestions, they use it as a sounding board rather than an oracle. Others have learned something different: how to work AI output until it produces the kind of answer that satisfies the rubric.
Both groups use the same tools. They’re learning completely different things about power.
The difference isn’t the technology. It’s whether anyone—teacher, parent, administrator—has ever asked them to look critically at the systems that evaluate them. Whether anyone has said: this tool makes a decision about you. Do you know what it’s optimizing for? Do you agree with its criteria? Can you articulate what it gets wrong?
Most schools haven’t asked that question. The students heading to Boston this summer will come closer to asking it than most students ever will. But even they are being channeled toward policy design—guardrails, use cases, disclosure requirements—rather than the more uncomfortable question underneath: should your educational experience be organized around systems that evaluate you without knowing you?
The Decision We’re Already Making
There’s a version of the next twenty years where a generation grows up accepting AI-mediated assessment as normal, natural, and beyond question. Where “the system gave me a B” has fully replaced “my teacher graded me a B,” and where the difference between those sentences—the human relationship in the second one, the appeal-ability of it, the right to say “that rubric missed something about what I was trying to do”—is simply gone.
And there’s a version where the next generation grows up fluent not just in using AI, but in interrogating it. Knowing how to ask: who built this? What were they optimizing for? Whose values are embedded in this model’s definition of good work?
We’re choosing between those two versions right now. Not in a single policy decision—but in thousands of small choices about how AI tools get deployed, what students are asked to do with them, and whether anyone teaches students to see the difference between performing within a system and questioning whether the system is right.
The students in Boston this summer are getting a chance to weigh in on the rules, and that’s worth taking seriously. What we still haven’t done is ask them the bigger question.
Takeaway for Teachers
Pick one AI tool your students use regularly. Have them spend 15 minutes this week reverse-engineering it: What does it reward? What does it penalize? Who designed it, and what were they optimizing for? You don’t have to abandon the tool—just make sure your students can see it.
A Deeper Dive
If you’re thinking through questions like this one—what AI in schools is teaching students about power, authority, and their own agency—my book goes deeper. The AI Doesn’t Know Your Students is available on Amazon and at shouldiuse.ai/book.
David Jacobson is a high school history teacher. He writes about AI, education, and the messy intersection of the two at shouldiuse.ai.
