The Impact of Chatbots on Students' Reflective Thinking in Introductory Programming Course

This study investigated New Zealand students' views on utilising chatbots and the impact of using ChatGPT or similar chatbots with suggested prompts on the reflective thinking skills of university students. The research was conducted in an introductory computer programming course (COMP102) in the School of Engineering and Computer Science. This introductory course uses a flipped classroom approach. The students were introduced to an approach to use a generative AI-based chatbot as an intervention. The intervention consisted of a brief explanation of chatbot training and utilisation, along with a specifically designed prompt intended to support learning. An example of prompt usage was demonstrated. Students' feedback was collected through two instruments: a Reflective Thinking Scale questionnaire, which measured students' reflective thinking abilities, and an open-ended student guide questionnaire designed to gather qualitative insights into their views and experiences using chatbots. Employing a mixed-methods research approach, we incorporated quantitative (Reflective Thinking Scale questionnaire) and qualitative (open-ended student guide questionnaire) data collection instruments within an experimental pre-test and post-test control group design. The pretest yielded 86 completed responses, while the post-test had 49 participants out of over 400 students enrolled in the course. The pre-test and posttest results did not yield quantitatively significant results in students' reflective thinking after the intervention. However, 21 out of 29 qualitative post-test respondents who utilise chatbots indicated that their interactions with the chatbot showed evidence of reflective thinking that improved their learning processes. The results revealed that students initially had doubts about using a chatbot for their learning, as they felt it might hinder their learning process or be unnecessary. Students perceived chatbots to be unreliable due to providing wrong answers, inaccurate code snippets, hallucinations, and plagiarised content. Therefore, students preferred to be self-reliant and not dependent on chatbots. Following the intervention, students demonstrated an increase of confidence in utilising chatbots as a learning tool, especially those who also used the suggested prompt. Students found that chatbots and the suggested prompt could be used to support learning and not only to provide answers. The chatbot was beneficial as a learning tool, providing additional information to clarify their understanding and reasoning behind providing simple explanations of programming concepts. Future work will focus on an intervention study in a more controlled environment to isolate the features of generative AI tools' impact and minimise external factors.