Zola 2025-09-10T23:56:27+00:00 https://notes.rcr.pub/tags/blog/atom.xml 2025-09-10T23:56:27+00:00 2025-09-10T23:56:27+00:00 Human https://notes.rcr.pub/posts/2023-era-ai-tutor-study-in-nature/ <p>A recurring problem with academic AI studies has been the severe lag time. At best, articles seem to have been published with one or two year lags, and since the AI state of the art has been moving on a monthly basis many results have been largely irrelevant. GPT 4 released on 2023-03-14, and now is finally showing up in peer reviewed journals. Published in Nature 2025-06-03 was an article from Harvard researchers studying the efficacy of a particular AI tutor chatbot in comparison to recent evidence based 'active learning' classroom techniques. In short, they found extremely strong evidence that their use of the AI tutor substantially improved the quality of the education while being more engaging to the learner.</p> <p>The authors don't attempt to overstate their findings however, and qualifying what exactly they claim is necessary to avoid misrepresentation by omission. The study was done on the Harvard course "Physical Sciences 2", specifically lessons on surface tension and fluid flow. The scope was carefully limited to reduce confounding variables, such as prior and connected knowledge, while capping the higher order goals at analysis, application, and understanding to the exclusion of synthesis and evaluation (according to Bloom's taxonomy). With this limited scope they then dedicated extensive resources to iterating on the LLM prompts and exercises to handle the strengths and weaknesses of GPT 4 specifically. They specifically used GPT-4-0613, which had a context window of only 8192 tokens. For context, the system prompt they provide was itself 601 tokens and the model could only handle one problem at a time. The exercises themselves were highly structured and linear; the hallucination problem was deemed, at the time, effectively insurmountable and hence the prompt for each exercise included a step by step walkthrough of the exercise, with strict instructions to only disclose one step at a time on request. The following is a Markdown and ASCII adaptation of an example exercise prompt, minus the global prompt all exercises had in common.</p> <hr /> <h2 id="question-specific-prompt"><a class="zola-anchor" href="#question-specific-prompt" aria-label="Anchor link for: question-specific-prompt"><i class="icon"></i></a> Question-Specific Prompt</h2> <h3 id="problem"><a class="zola-anchor" href="#problem" aria-label="Anchor link for: problem"><i class="icon"></i></a> PROBLEM:</h3> <p>Vorticella is a stalk-shaped protozoan, which contracts rapidly. Suppose you wish to study the flow about the cell body keeping the Reynolds number the same but with a model that is scaled up by a factor of 100. How much slower or faster must the model move the body, compared to the speed of the real vorticella?</p> <h3 id="answer"><a class="zola-anchor" href="#answer" aria-label="Anchor link for: answer"><i class="icon"></i></a> ANSWER:</h3> <p>The model must move 100 times slower than the real vorticella.</p> <h2 id="keep-in-mind"><a class="zola-anchor" href="#keep-in-mind" aria-label="Anchor link for: keep-in-mind"><i class="icon"></i></a> Keep in mind:</h2> <p>Steps</p> <ul> <li>Set the Reynolds numbers equal to each other. <ul> <li><code>Re_real = Re_model</code></li> </ul> </li> <li>Put in the expression for Reynolds number assuming both the real vorticella and model are in water (denoted "<code>w</code>"). <ul> <li><code>(ρ_w * l_real * v_real) / η_w = (ρ_w * l_model * v_model) / η_w</code></li> </ul> </li> <li>Since both sides have the density of water (<code>ρ_w</code>) and viscosity of water (<code>η_w</code>), we can cancel those out on both sides. <ul> <li><code>l_real * v_real = l_model * v_model</code></li> </ul> </li> <li>Solve for the model's velocity (<code>v_model</code>). <ul> <li><code>v_model = (l_real / l_model) * v_real = (1/100) * v_real</code></li> </ul> </li> <li>So, the model of the vorticella must move 100 times slower than the real vorticella to get the same Reynolds number and achieve Dynamic Similarity.</li> </ul> <hr /> <p>Essentially, the use case gave the model every opportunity to suceed while mitigating its known weaknesses. The emphasis on iteration on these prompts is key: naive attempts without the step by step instructions or strict linearity would have likely, at best, had inconsistent model response quality. Not to imply the study is 'bad', but rather should be understood as an early proof of concept or demonstration where the task and tool were coevolved to get the best results as opposed to applying the tool to an independent task. There are further confounding variables, such as the student population of Harvard in particular, novelty of the technology at the time, or behavior of the instructors, but that's getting beyond the point.</p> <p>The point is the strength of this evidence; they calculated 0.73 to 1.3 standard deviations of improvement over the active learning classroom method with a p value less than 10^-8. That degree of confidence for a topic as messy as teaching methods made me double check if that was still the p value that usually just has to be p &lt; 0.05. That's very strong evidence, albeit specific to this heavily engineered scenario. That said, the developments in LLMs since the study was conducted invites some very consequential questions and corresponding opportunities. The study's scope and shape was carefully shaped by the contours of GPT 4's limitations, but current generation models have contexts in the hundreds of thousands to million(s) of tokens, much stronger instruction following and recall, and some ability to emulate reasoning through complex problems. Or simply, many of the defining limitations are now gone.</p> <p>So, why is this important and what now? I can't escape a feeling of urgency implied by these results, in that educational institutions, arguably even the learners themselves, have a duty to the students (or in the students' case, themselves) and society to use the best evidence based methods available. At the same time, it's becoming increasingly clear (or rather likely to be true) that chronic use of LLMs uncritically can effectively become a crutch and allow the very faculties meant to be improved to instead atrophy. This expands very quickly to socioeconomic and geopolitical levels too: what if a group or nation adopts teaching methods and technologies that effectively bump the graduated students' abilites a whole standard deviation, by whatever metric, or to graduate the same students in 25-50% less time? As for what now, if there is going to be a substantial lag between capabilities and confident study of those capabilities, then teaching meta level skills seems a flexible and durable way to manage. Rather than making them as human and invisible as possible, perhaps rather demystify and dissect, allowing people the ability to make informed decisions about how to use them on their own, how to combat the cognitive biases that creep in, or how to measure performance in the first place.</p> <h2 id="links"><a class="zola-anchor" href="#links" aria-label="Anchor link for: links"><i class="icon"></i></a> Links</h2> <hr /> <h2 id="nature-articlesupplementary-data-documentation"><a class="zola-anchor" href="#nature-articlesupplementary-data-documentation" aria-label="Anchor link for: nature-articlesupplementary-data-documentation"><i class="icon"></i></a> <a rel="external" href="https://www.nature.com/articles/s41598-025-97652-6">Nature article</a><br /> <a rel="external" href="https://static-content.springer.com/esm/art%3A10.1038%2Fs41598-025-97652-6/MediaObjects/41598_2025_97652_MOESM1_ESM.docx">Supplementary data/documentation</a></h2> 2025-08-24T22:23:50+00:00 2025-08-24T22:23:50+00:00 Human https://notes.rcr.pub/posts/mlp-visualizer/ <h2 id="mlp-visualizer"><a class="zola-anchor" href="#mlp-visualizer" aria-label="Anchor link for: mlp-visualizer"><i class="icon"></i></a> <a href="/mlp-vis/index.html">MLP Visualizer</a></h2> <p><a href="/mlp-vis/index.html"><img src="../2025-08-24-mlp-visualizer.png" alt="MLP Visualizer preview" /></a> I finished a project I've been working on for awhile that is meant to visualize multi-layer perceptrons (MLPs) for educational purposes, found <a href="/mlp-vis/index.html">here</a>. The project was mostly written by AI, with the original self contained html page being written in a chat interface and the deployed extension by Claude Code. Overall, this took a couple hours of actual work for me with the AI doing the vast majority of the actual work. It cost about $45 in full, having used Sonnet 4 as the main model. It's certainly not perfect, but I think it's good enough to be put into the wild.</p> <p>The tech stack I specified was basically just a Nix flake to manage the project, npm for depsendencies, Vite to build, Svelte for the front end framework, and Bulma for themeing. Axe was also used later on for accessibility testing. Otherwise, I mostly have no idea how it works under the hood. I know it uses canvases to performantly draw the graphs and uses a decoupled simulate and draw loop, but other than that I don't know much. I'm not particularly keen on some of the architectural choices, with a lot of core login being in App.svelte instead of in Typescript, but then again I never had to deal with it.</p> <p>I wasn't as careful with this project as some of the bigger ones, and I think that shows. Instead of having a large pile of documentation and logging to keep the model focused it was very apparent when the context was lost. Sometimes it'd forget how to build and run the app, other times it'd seem like it was rediscovering what the app essentially was (not to anthropmorphize, that was essentially true without a log). On the other hand, this made it closer to the 'vibe coding' some talk about, where I lazily iterate through high level ideas while half paying attention.</p> <p>Ultimately, if I more carefully documented and planned this from the start, the result would probably be both better and cheaper. But that's the devil's bargain at play, you get to trade money for time and effort.</p>