The AI doesn’t need to be told who your students are. The platform handed it the data months ago.
On December 28, 2024, PowerSchool disclosed that a hacker had downloaded the records of 62 million students and 9.5 million educators from its customer database. The data included names, birthdates, addresses, Social Security numbers, medical conditions, disability accommodations, individualized education plans, disciplinary records, and family income tied to free and reduced lunch eligibility. It was the largest breach of children’s data in US history.
The breach was preventable. PowerSchool didn’t require multi-factor authentication on its support portal. Industry standards have treated MFA as baseline for over a decade. The hacker was a 19-year-old college student in Massachusetts who used stolen credentials and a customer-support tool to walk straight in.
Why was 62 million students’ worth of that data sitting on PowerSchool’s servers in the first place?
The answer is that to operate a school information system, you need to collect that data. Names and addresses for parent contact. Birthdays for grade-level reports. IEPs for accommodation. Income data because the federal free-and-reduced-lunch program requires it. Discipline records for compliance. SIS platforms collect it because schools need it, and schools store it on SIS platforms because that’s the architecture.
Then last July, Instructure announced a global partnership with OpenAI to embed large language model capability directly into Canvas, the LMS used by more than thirty million students. AI-graded assignments, AI feedback on student work, AI summaries of student progress: all running inside the platform, with full access to the platform’s data.
Together, those two developments created a condition most schools haven’t mapped.
The platform-level inference problem
Earlier this week I wrote about a Stanford study showing that AI feedback shifts when researchers tell the model a student’s race, gender, learning needs, or motivation level. That finding has a simple practical response: don’t tell the AI who the student is.
That works fine for a teacher who pastes a student essay into ChatGPT in a fresh window.
It works less well when the AI is embedded in a platform that already knows the student’s IEP status, free-and-reduced-lunch flag, prior grade history, and disciplinary record. The teacher doesn’t have to type any of that into a prompt. The platform routes it through automatically.
You can imagine, without anything especially paranoid, a Canvas-integrated AI feedback tool that pulls a student’s IEP accommodations to “personalize” the feedback it gives. The Stanford finding suggests what that personalization will look like: extra praise for students marked by disability, less substantive critique. The teacher never saw a prompt reading “this student has an IEP.” The platform handled that automatically.
The Stanford researchers noted that even without explicit demographic labels, modern LLMs can infer aspects of student identity from writing patterns. The platform integration removes the inference step. The data is already on the wire.
Three Questions for Your Administration
The LMS layer is where individual teachers have less say than usual. The integration decisions get made above your head.
Three questions your administration should be able to answer before the next AI feature ships:
What data does the AI feature have access to by default? If “the assignment text plus the rubric” is the answer, ask for that to be documented. If the answer is broader, covering IEP flags, demographic data, or prior performance, ask why that data is included and what audit is in place.
Has the vendor published a bias audit? That means the same assignment scored under different student demographic conditions, as in the Stanford study. If the vendor can’t produce one or won’t, that’s the answer.
What’s the off-switch? Schools accept opt-out for many things. AI feedback embedded in the LMS should be one of them, at the district level (do we turn this feature on?) and at the teacher level (do I use it for this assignment?).
None of these questions require deep technical knowledge. They require asking the vendor’s representative to be specific, and being willing to take “we don’t have that information yet” as a real answer.
Platform Data and Teacher Judgment
The premise of my book is that the AI doesn’t know your students. The conversations after class, the read on what a kid is and isn’t ready for, the knowledge that comes from being in the room with them: none of that is in the model, and it can’t be prompted in.
The PowerSchool breach and the Canvas integration are the structural counterpoint to that argument. The AI doesn’t know your students, but the platform you’ve been instructed to use does, and the AI now has access to what the platform knows. The teacher’s read on a student remains uncopyable. The platform’s read on a student is what the AI draws on in the teacher’s absence.
The book works through that gap across every layer of classroom practice: what teachers can do that no platform can replicate, where vendors are positioning their tools to imitate it, and what assignment-level decisions remain meaningful when the platform decisions get made elsewhere.
Check Your LMS This Week
Pull up your LMS this week. Find the “data and privacy” or “integrations” section in the settings. Look at what data is being routed where, and which third-party tools have access. If the LMS has rolled out an AI feature in the last six months, find the page that lists what data that feature can read.
Nothing here requires immediate action. Knowing what’s flowing through the system before the next vendor presentation is enough.
If you’re working through these questions about what your LMS routes to AI features and what teachers can ask before the next integration ships, my book goes deeper. The AI Doesn’t Know Your Students is available on Amazon and at shouldiuse.ai/book.
David Jacobson is a high school history teacher in Shanghai. He writes about AI and education at shouldiuse.ai and is the author of The AI Doesn’t Know Your Students.
