Evaluation of Enhanced Programming Error Messages Generated by Large Language Models from Instructors' Point of View

Enhanced Programming Error Messages are one type of feedback that may be useful during the learning process of novice programmers. One way to improve such messages is by using the context given by an auto-grader from students' errors when solving programming exercises in those environments. We investigate how Large Language Models (LLM) may be used to provide additional feedback beyond simple compiler or interpreter error messages by garnering input from a programming exercise description, the student's code to solve that exercise, and the error generated by a Python interpreter and generating an LLM prompt from that given information. In an experiment, we asked instructors to evaluate the prompt text generated by an LLM from a variety of exercises and the students' errors. Our results show a high rate of adequate feedback by means of explanations and code examples to fix the errors without providing a full code solution. However, some cons have also been noticed, such as vague explanations, hallucinations, and dissociation from explanations and examples. The results suggest that LLM feedback is a promising approach to provide feedback to novice programmers by means of enhanced programming error messages.