Embracing the challenge: Predicting self-testing in non-formal online courses using machine learning

Technology-driven advancements have made adaptive and interactive learning techniques more accessible. Online courses increasingly integrate self-tests that offer automated and immediate feedback. Self-tests can help learners to identify knowledge gaps and to reinforce their retention and comprehension. However, not all learners readily use self-tests, raising the question of which factors may impact learners’ engagement with self-tests. The present study focused on non-formal education, covering 45 online courses offered by Bavarian universities. Analyses were based on a sample of N =1,261 participants aged 16 to 84 years. We used a machine learning approach to predict learners’ engagement with self-tests and to identify important influencing factors. Therefore, we included 50 predictor variables in an elastic net regression to explore the role of learner- and course-related characteristics. The predictor variables were drawn from self-report, process, and meta data. Overall, learners differed substantially in their self-testing behavior. The prediction model explained 11% of the variance in learners’ engagement with self-tests. Despite the model’s modest explanatory power, the analysis identified potentially relevant predictors. The two most important predictor variables were learner commitment and the intention to obtain a confirmation of participation. Accordingly, course designers might implement extrinsic incentives—such as confirmations of participation—as a potentially useful strategy to encourage learners’ engagement with self-tests. From a methodological perspective, the study highlights the importance of using appropriate statistical methods—such as machine learning algorithms—to understand complex learning behaviors. • Learners differ in their engagement with self-tests in non-formal online courses • Findings highlight the importance of commitment and extrinsic incentives • Enrollment intentions play a central role in the prediction of self-testing • Socio-demographic information did not predict the engagement with self-tests • Understanding complex learning behavior requires appropriate analytical methods