Investigating Validity and Generalisability in Trace-Based Measurement of Self-Regulated Learning: A Multidisciplinary Study

Self-regulated learning (SRL) skills are critical for effective learning and academic success. With the growing availability of trace data from students' online learning activities, researchers are increasingly leveraging this data to infer SRL processes. However, challenges remain regarding the validity of these inferences and their generalisability across diverse learning contexts. This study presents a structured approach to investigate these challenges by examining SRL behaviours in a multidisciplinary university cohort. The dataset includes 76 baseline survey responses, over 300 daily SRL survey submissions, and more than 6,000 sequences of recorded learning actions as trace data. Using mixed linear models and sequence mining, the analysis is grounded in SRL theory and evaluated through machine learning performance metrics. Our findings indicate consistent within-person patterns of SRL and online learning behaviours, supporting the concept of transferable, holistic skill development. Additionally, the results validate the trace-based detection of SRL engagement but highlight limitations in accurately detecting planning and reflection phases. These findings underscore the potential of automating SRL engagement detection while emphasising the need for multi-modal approaches to capture the full spectrum of SRL processes comprehensively.