Applying Machine Learning Techniques on Self-Reported Engagement and Student Log Data to Predict CS Learning Performance

Enhancing student engagement in computer science (CS) courses is crucial to improving students' achievement and fostering active participation in computer science education (CSE). Previous studies have highlighted different factors that shape student engagement, including behavioural, cognitive, emotional, and social engagement. Additionally, other factors influence student engagement, such as students' beliefs in the usefulness of learning computer science and their confidence in taking CS classes. Despite existing studies investigating student engagement in CSE, limited studies have explored factors that influence novice student engagement in CS courses. Further, no study has applied machine learning (ML) techniques on the combined self-reported engagement data and student log data to predict CS learning performance. Therefore, this study used ML techniques to explore and identify student engagement factors that affect and predict CS learning outcomes. To achieve this, data was collected using three different sources: self-reported data, system log data, and CS learning performance data. Log data from student behaviour on the system included monthly logs of student access to the CS course page on the LMS during the semester, the total hits of student interactions with the course content throughout the semester, and the total number of task submissions. Our analysis involves 77 novice students who consented and completed a multidimensional self-reported questionnaire during the second semester of 2022 - 2023 at a university in Saudi Arabia. The K-means clustering algorithm was used to understand engagement patterns by classifying students into groups based on their levels of self-reported engagement, log data, and academic performance. Classification algorithms using Random Forest (RF), Decision Tree (DT), and LightGBM (LGBM) were used to predict CS learning performance from student engagement data (self-reported and logs). We evaluated the performance of ML algorithms using metrics including accuracy, precision, recall, and Fl-score. The clustering results showed that students who actively engage with the course content (log data) tend to achieve higher grades, especially those with higher total hits of student interactions with the course content throughout the semester. The classification results showed that the RF model outperforms DT and LGBM, highlighting the significance of student interactions with the course in the first month of the semester as the key indicator influencing CS learning performance. Our findings contribute to a deeper understanding of student engagement in CS education and highlight various sources and factors used to measure and influence student engagement. The study has implications for educators, researchers, and stakeholders who may design effective interventions that would increase engagement to improve student learning outcomes in CS education. Future work will use a larger sample of participants from various educational levels in different countries.