Improving the Efficacy of an Automated Judging System in an Introductory Programming Class using a Large Language Model

There is evidence supporting that extensive practice in programming learning improves learning outcomes. Automated judging systems are commonly used as a tool to allow programming students to have access to a vast pool of exercises to improve their skills. These systems help students by assessing the correctness of their solutions. However, the feedback received is binary and limited to indicating whether the solution is correct, depending on whether the program passes all the provided test cases. Also, creating new exercises and test cases remains a demanding task for instructors.Large Language Models have recently gained popularity due to their remarkable ability to learn and generate highly proficient human language. These systems are being used in various contexts, including education.This paper presents our efforts to extend the automated judging system used in the Introduction to Programming class at the Universidad Técnica Federico Santa María in Chile. We used a commercial system supported by a Large Language Model to incorporate new functionalities into our platform. The extended system can create new programming exercises for each unit in the course and guide students’ learning by recommending exercises based on their demonstrated performance. It also provides enhanced feedback on students’ solutions to help them identify and correct their errors. In the early stages of problem-solving, the system asks students questions to ensure they fully understand the problem statement and the desired behavior of the solution. Additionally, the platform collects information that allows for a deeper analysis of students’ common mistakes and learning patterns.A preliminary evaluation of the extended system’s features shows promising results regarding the effectiveness of the mechanisms for exercise generation, guided learning, automatic verification of problem understanding, and enhanced feedback. An improvement in students’ self-efficacy was also observed.