Assessing the Impact of Integrating ChatGPT as an Advice Generator in Educational Software

This paper reports on the study of the integration of ChatGPT as an advice generator in custom educational software developed for Java programming. The software, in cooperation with ChatGPT API, pursues providing real-time, context-specific advice to students for better learning. This work adopted a two-fold evaluation approach to evaluating this integration. First, this study examines the effectiveness of this integration with the help of the Interrupted Time Series Analysis methodology to measure possible improvement in the performance metrics of the students in terms of error rates and task completion times. Second, this work presents a custom-designed questionnaire used to get student perceptions regarding the clarity, usefulness, and impact of ChatGPT’s advice, and the level of student satisfaction with the user interface. The key takeaways from this research study are the substantial improvements in performance metrics that were noted quantitatively, with students achieving lower error rates and faster completion times after the intervention of ChatGPT. Qualitatively, learners express their satisfaction with the clarity of advice, which gives them an understanding that works on their learning and confidence in Java programming. These findings point toward the promise of integrating such advanced AI solutions in educational software toward a significant improvement in learning outcomes and the necessity of human-aided continuous user feedback for system refinement.