It isn’t what you have. It’s how you use it.
A recent large‑scale study on AI tutors shows how using AI differently, makes a difference.
Much of the traditional AI support centres on students asking a question and the AI tutor responds.
The researchers argue this is not enough for real learning, because:
→ Students often don’t know what to ask
→ They ask for answers too quickly
→ They struggle to regulate their own learning
→ The right level of challenge is missing
In their research they tested whether AI tutors can improve learning outcomes by proactively choosing the right next practice problem for each student, moment by moment, based on how that student is actually learning, not just because they got answers right.
They incorporated three elements:
1. A socratic chatbot that encourages effort, and hints without answering
2. A process to customise the next step for the student
3. A model that watches HOW the student learns and adapts, not just WHAT.
They ran a large, long‑duration randomized controlled trial with 770 high‑school students, 10 schools, and a 5‑month Python programming course
The final assessment was in‑person, written, with no AI allowed
Students were randomly assigned to the Control group with a fixed sequence of problems to solve, or the Treatment group with personalized sequence chosen by the AI system
The result? Students with personalized problem sequencing scored ~0.15 standard deviations higher on the final exam. Equivalent to 6–9 months of additional schooling. An impressive feat.
These gains happened without more instruction time, without extra teacher workload, without giving students more problems to solve.
The students with least Python experience benefited the most.
The improvements were due to students spending more time productively working, they were engaged. They persisted longer on problems and their interactions with the AI tutor were higher quality with more exploration and curiousity.
It is a good reminder, that it isn't what you use (GenAI), it is how you use it.
SOURCE
