It’s Dangerous to Prompt Alone! Exploring How Fine-Tuning GPT-4o Affects Novices’ Programming Error Resolution

Large language models (LLMs) like GPT-4o are increasingly being incorporated into computer science classrooms in tasks such as helping students resolve programming error messages. Prior work has found weak or insignificant support that LLM-generated error explanations help students resolve errors faster, especially compared to expert-handwritten error explanations. This work explores how GPT-4o can be influenced through fine-tuning to produce error explanations that resemble expert-handwritten explanations. We evaluate how three error message styles (traditional Python error messages, baseline GPT-4o explanations, fine-tuned GPT-4o explanations) affect novice programmers’ ability to resolve programming errors. Eighty (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> = 80) CS1 students completed six debugging tasks using a within-subjects study design that ensured each participant saw each of the three error message styles twice. We measured time-to-fix, number of code reruns, task failure rate, and then asked for students’ preferences. Participants were found to resolve errors significantly faster with the fine-tuned GPT-4o messages, but, like prior work, we could not find a statistically significant difference between the baseline GPT-4o and traditional Python error messages. Error message style had no effect on number of reruns and task failures; however, the particular debugging task had a significant difference on both. Despite solving problems faster with the fine-tuned GPT-4o model, participants preferred the baseline GPT-4o output (similar to unmodified ChatGPT). These findings strongly suggest that the nature of the debugging task has a greater effect on novice programmers’ error resolution than whether or not the students use a GenAI system to help them debug.