2023 Era AI Tutor Study in Nature
By Human • 6 minutes read •
A recurring problem with academic AI studies has been the severe lag time. At best, articles seem to have been published with one or two year lags, and since the AI state of the art has been moving on a monthly basis many results have been largely irrelevant. GPT 4 released on 2023-03-14, and now is finally showing up in peer reviewed journals. Published in Nature 2025-06-03 was an article from Harvard researchers studying the efficacy of a particular AI tutor chatbot in comparison to recent evidence based 'active learning' classroom techniques. In short, they found extremely strong evidence that their use of the AI tutor substantially improved the quality of the education while being more engaging to the learner.
The authors don't attempt to overstate their findings however, and qualifying what exactly they claim is necessary to avoid misrepresentation by omission. The study was done on the Harvard course "Physical Sciences 2", specifically lessons on surface tension and fluid flow. The scope was carefully limited to reduce confounding variables, such as prior and connected knowledge, while capping the higher order goals at analysis, application, and understanding to the exclusion of synthesis and evaluation (according to Bloom's taxonomy). With this limited scope they then dedicated extensive resources to iterating on the LLM prompts and exercises to handle the strengths and weaknesses of GPT 4 specifically. They specifically used GPT-4-0613, which had a context window of only 8192 tokens. For context, the system prompt they provide was itself 601 tokens and the model could only handle one problem at a time. The exercises themselves were highly structured and linear; the hallucination problem was deemed, at the time, effectively insurmountable and hence the prompt for each exercise included a step by step walkthrough of the exercise, with strict instructions to only disclose one step at a time on request. The following is a Markdown and ASCII adaptation of an example exercise prompt, minus the global prompt all exercises had in common.
Question-Specific Prompt
PROBLEM:
Vorticella is a stalk-shaped protozoan, which contracts rapidly. Suppose you wish to study the flow about the cell body keeping the Reynolds number the same but with a model that is scaled up by a factor of 100. How much slower or faster must the model move the body, compared to the speed of the real vorticella?
ANSWER:
The model must move 100 times slower than the real vorticella.
Keep in mind:
Steps
- Set the Reynolds numbers equal to each other.
Re_real = Re_model
- Put in the expression for Reynolds number assuming both the real vorticella and model are in water (denoted "
w").(ρ_w * l_real * v_real) / η_w = (ρ_w * l_model * v_model) / η_w
- Since both sides have the density of water (
ρ_w) and viscosity of water (η_w), we can cancel those out on both sides.l_real * v_real = l_model * v_model
- Solve for the model's velocity (
v_model).v_model = (l_real / l_model) * v_real = (1/100) * v_real
- So, the model of the vorticella must move 100 times slower than the real vorticella to get the same Reynolds number and achieve Dynamic Similarity.
Essentially, the use case gave the model every opportunity to suceed while mitigating its known weaknesses. The emphasis on iteration on these prompts is key: naive attempts without the step by step instructions or strict linearity would have likely, at best, had inconsistent model response quality. Not to imply the study is 'bad', but rather should be understood as an early proof of concept or demonstration where the task and tool were coevolved to get the best results as opposed to applying the tool to an independent task. There are further confounding variables, such as the student population of Harvard in particular, novelty of the technology at the time, or behavior of the instructors, but that's getting beyond the point.
The point is the strength of this evidence; they calculated 0.73 to 1.3 standard deviations of improvement over the active learning classroom method with a p value less than 10^-8. That degree of confidence for a topic as messy as teaching methods made me double check if that was still the p value that usually just has to be p < 0.05. That's very strong evidence, albeit specific to this heavily engineered scenario. That said, the developments in LLMs since the study was conducted invites some very consequential questions and corresponding opportunities. The study's scope and shape was carefully shaped by the contours of GPT 4's limitations, but current generation models have contexts in the hundreds of thousands to million(s) of tokens, much stronger instruction following and recall, and some ability to emulate reasoning through complex problems. Or simply, many of the defining limitations are now gone.
So, why is this important and what now? I can't escape a feeling of urgency implied by these results, in that educational institutions, arguably even the learners themselves, have a duty to the students (or in the students' case, themselves) and society to use the best evidence based methods available. At the same time, it's becoming increasingly clear (or rather likely to be true) that chronic use of LLMs uncritically can effectively become a crutch and allow the very faculties meant to be improved to instead atrophy. This expands very quickly to socioeconomic and geopolitical levels too: what if a group or nation adopts teaching methods and technologies that effectively bump the graduated students' abilites a whole standard deviation, by whatever metric, or to graduate the same students in 25-50% less time? As for what now, if there is going to be a substantial lag between capabilities and confident study of those capabilities, then teaching meta level skills seems a flexible and durable way to manage. Rather than making them as human and invisible as possible, perhaps rather demystify and dissect, allowing people the ability to make informed decisions about how to use them on their own, how to combat the cognitive biases that creep in, or how to measure performance in the first place.