We present an article written by our Featured Guest Expert Ralph T O'Brien for our school and multi academy trust customers about using AI in schools and trusts. Ralph O’Brien is a global privacy and data protection professional with over 25 years’ experience helping organisations manage data protection and privacy risk, prevent data harms, and build practical, trustworthy information governance programmes

Artificial Intelligence is shorthand for more automation.  Less humans, more machines.  AI is already in schools, and not because the headteacher has approved an AI strategy. Not because the governors have discussed it.

It is there because Teachers, Pupils and Parents are already using it - and even if they didn’t want to.  The software companies already uses it and package it within the Technology. That creates a problem.

The debate about AI in education has often become a discussion of two extremes:

A: embrace AI because it will transform education, 

B: resist it because it creates unacceptable risks.

Neither will work. But to do otherwise involves doing some work.  They need to understand where AI creates genuine educational benefit, where it creates harm, and what needs to be done to make the first more likely and the second less likely.

Ralph O’Brien, of Serious Privacy Ltd, has worked with Data Protection Education to create a framework of six Pillars.

Purpose. People. Protection. Product. Practice. Proof.

1. Purpose

The first question shouldn't be: "Is this AI tool GDPR compliant?", but "What problem are we trying to solve, and have we been given a choice?"

AI is being promoted as a solution to everything from teacher workload and lesson planning to personalised learning, assessment, attendance and safeguarding.  Essentially AI is more of a commercial marketing tool than anything else.  Tech companies are scared of being left behind in the “AI revolution” throwing it into their products to be competitive, regardless of environmental and societal impact.

But the availability of a technology doesn't mean that it is the right technology for a particular problem.

Suppose a school wants to introduce an AI system to identify pupils who may be falling behind. Sure, But Why? Perhaps the objective is to identify children who need additional support early, potentially a worthwhile objective.

But what evidence suggests that the AI will do this effectively? Could the same outcome be achieved through a less intrusive approach? What happens if the system systematically identifies some children as "at risk" when they aren't? What happens if it fails to identify children who genuinely need help?

And perhaps most importantly: What does good look like?

Any AI adoption shouldn't begin with a product demonstration, instead it should begin with a clear educational purpose.

2. People

An AI system may affect:

  pupils;

  teachers;

  teaching assistants;

  safeguarding teams;

  parents;

  school leaders;

  governors and trustees;

  applicants and prospective pupils;

  and people who may never directly interact with the system at all.

And naturally children require particular attention, being less aware of the risks.

Teachers may assume that the system is intelligent because it speaks confidently and fluently, and children may assume that it is trustworthy because their teacher has allowed them to use it.

We are seeing many people disclosing information that they would never consciously provide to a conventional database or treating an AI-generated answer as authoritative when it is simply sounding confident, forgetting it is simply a probability engine, returning answers on a statistical basis from the data.

This matters because the relationship between a child and an AI system isn't necessarily equivalent to the relationship between an adult and a piece of workplace software.

There can be differences in understanding, power, maturity and vulnerability and human relationships are at stake.  If a pupil is struggling, should the first response be an AI chatbot?  If a child is anxious, should they be encouraged to discuss that anxiety with an AI?  If a pupil receives feedback about their ability, should that feedback come from a statistical model?

Technology can support human relationships, where we are in danger of replacing them.

3. Protection

This is where data protection professionals have an important role, but not in just producing compliance documentation.  When schools consider AI, the traditional data questions remain important:

  What data is being processed?

  Is it lawful, in basis and use of training data?

  Is special category data involved?

  Where is the data processed?

  Who are the controllers and processors?

  What sub-processors are involved?

  How long is information retained?

  What information is used to train models?

  What transparency is provided?

  Has a DPIA produced real protection measures

AI can create risks even when the underlying data processing looks perfectly manageable, such as taking ordinary pieces of information and use them to generate inferences – which may be wrong. It might influence how teachers perceive the child.  And potentially follow the child through their educational journey.

So we need to ask not just: "What data are we protecting?"

But instead: "What could happen to this child because of the way the data is being used?"

A technically compliant system can still produce poor outcomes.  Sometimes we need to forget about producing documents and think about actual measures we can bring in to achieve real human protections.

4. Product

Schools buy products from vendors. Those vendors are pushing AI.  Schools have no control over that inclusion.  Do we really understand what is happening inside those products?

A supplier saying their product is GDPR compliant, doth butter no parsnips.

Schools need to know:

  What AI model is being used?

  What information does it receive?

  What information does it retain?

  Is data used for training?

  Where does processing take place?

  Who else receives the information?

  What happens when the model changes?

  How is accuracy assessed?

  How is bias tested?

  What safeguards exist for children?

  Can the school control how the system is used?

  Can a pupil or teacher challenge an output?

  What happens when the system is wrong?

And they don’t have the resources or knowledge or time to ask those questions. So they simply accept that AI is becoming embedded into products that schools already use.  Even existing products may have acquired new AI functionality.

Schools need AI-aware procurement and supplier assurance, not simply another tick box in an IT procurement exercise.

5. Practice

A policy saying "staff must not enter personal data into public AI tools" is just not realistic anymore, AI is too prevalent.   Which means Schools need to understand behaviours of staff and pupils alike.

Teachers are under enormous pressure. If technology saves 30 minutes preparing a lesson, helps them to shortcut creating notes, or explain something they don’t know, someone will use it, and tech firms will make that option available and easy.

People will use it, but the question is whether they will use it well.

That means schools need practical guidance and training in AI literacy, and the same applies to pupils.

AI literacy is not how to get better answers from ChatGPT. But what AI can do, can't do, What information they should never input, and how to question outputs, how it can be wrong, how it can manipulate or persuade, and how training data optimised for the normal prevents the unusual and diverse from being represented – or even excluded altogether.

6. Proof

Schools are very good at demonstrating that something has been implemented.

We can produce; policies, DPIAs, contracts, training records, fair processing notices, procurement documentation. Whoop.  Companies will provide you all of that, but does that demonstrate real improvements and outcomes, or identifies the risks and drawbacks?

  If a school introduces an AI tutor, has it improved learning?

  If AI is introduced to reduce teacher workload, did it?

  If an AI system identifies pupils needing additional support, did it do it as well as a human?

  Does it work equally well for diverse and different groups of pupils?

  Are teachers appropriately challenging its recommendations?

  Are children using it safely?

  Are there unexpected harms?

  And what happens when the original assumptions turn out to be wrong?

Even documentation, like a DPIA, completed before deployment isn't the end of the process.  It is the beginning of accountability throughout its lifecycle.  AI models drift and change with new inputs and have a need for constant re-evaluation, as they make connections and inferences – and correlation and causation are not the same thing.

Bringing the Six Pillars Together

The six pillars aren't six separate subjects. They are “many to many” connected. And even if we implement these six basic pillars.  There remain the most important questions:

Can we turn it off? Can we challenge the outcomes? Can people still have agency? Can we re-establish the human?  Can we operate without it? Are we putting the human first?

There is a danger that AI governance in schools becomes another layer of bureaucracy.

Data protection professionals have something important to contribute here because they already understand concepts such as accountability, proportionality, purpose limitation, data minimisation and fundamental human rights.  But we have to be brought in early to make the difference.  Once the data is absorbed and the system is in place, it is often too late.

We don’t want to prevent schools from using AI, or to encourage schools to adopt it as quickly as possible.  We do need to help schools make good decisions about when, where and how AI should be used.

The question is whether schools can use technology in a way that serves humans, or if we are creating a world where humans are increasingly serving the technology?

Ralph has recently developed a Serious Privacy: Data Game, which your can register for the kickstarter at: https://www.kickstarter.com/projects/seriousprivacy/serious-privacy-the-data-game