A Multi-modal Understanding of Emotions and Cognitive Engagement of Students during a Programming Task

This study uses a multi-modal approach to investigate novice students' cognitive and emotional experiences during a programming task. We collected multi-modal data from twenty-eight students taking an introductory programming course for the first time. These data include eye-tracking and electrodermal activity (EDA) to assess cognitive engagement and emotional arousal in near real-time, followed by a retrospective think-aloud interview. Eye-tracking data were analyzed to identify students' focus within the programming environment, while peaks in the EDA data were used as a proxy for emotional arousal. By triangulating these data, we identified key programming events, such as debugging failure and syntax errors, influencing students' cognitive engagement and emotional responses. Preliminary findings suggest that students' prolonged focus and frequent transition between coding pane, terminal, and task description areas, particularly during debugging, align with peaks in emotional arousal. This may emphasize the role of task challenges in shaping students' experiences. Insights from this study could inform the need to integrate emotional and cognitive support into educational tools to personalize learning and provide near real-time support when students encounter difficulties. Having such tools could make programming courses more engaging and improve both student retention and learning outcomes.