Model Drift in Deployed Machine Learning Models for Predicting Learning Success

The use of learning success prediction models is increasingly becoming a part of practice in educational institutions. While recent studies have primarily focused on the development of predictive models, the issue of their temporal stability remains underrepresented in the literature. This issue is critical as model drift can significantly reduce the effectiveness of Learning Analytics applications in real-world educational contexts. This study aims to identify effective approaches for assessing the degradation of predictive models in Learning Analytics and to explore retraining strategies to address model drift. We assess model drift in deployed academic success prediction models using statistical analysis, machine learning, and Explainable Artificial Intelligence. The findings indicate that students’ Digital Profile data are relatively stable, and models trained on these data exhibit minimal model drift, which can be effectively mitigated through regular retraining on more recent data. In contrast, Digital Footprint data from the LMS show moderate levels of data drift, and the models trained on them significantly degrade over time. The most effective strategy for mitigating model degradation involved training a more conservative model and excluding features that exhibited SHAP loss drift. However, this approach did not yield substantial improvements in model performance.