Understanding Optimal Interactions Between Students and A Chatbot During A Programming Task

This study explores integrating Large Language Models (LLMs) into computer science education by examining undergraduate interactions with a GPT-4-based chatbot during a formative assignment in an introductory course. We aim to delineate optimal help-seeking behaviors and ascertain if effective problem-navigating strategies correlate with improved learning outcomes. Using descriptive statistics and Structural Topic Modeling (STM), we analyze the types of questions posed and their connection to task completion success. Findings reveal a positive association between the number of attempts and help requests, indicating more engaged students seek assistance. STM analysis shows high-ability students address abstract concepts early, while lower-ability students focus on syntax-related issues. These insights underscore the need to evaluate interaction behaviors to optimize chatbot use in education, leading to proposed guidelines to enhance chatbot utilization, promoting responsible use and maximizing educational advantages.