Reliability Coefficient for Bayesian Knowledge Tracing Models

Abstract Knowledge tracing (KT) refers to the process of efficiently tracking student achievement in online learning systems. Bayesian knowledge tracing (BKT) is a representative statistical model used for this process. Despite the widespread application of BKT in predicting student performance and modeling knowledge acquisition, methods to evaluate the consistency of measurement by BKT models remain elusive. In psychometrics, reliability is a fundamental concept that gauges the degree to which an assessment tool generates stable and consistent results. Evaluation of reliability is crucial because, without reliable measurement, the adequacy of educational assessments and the decisions based on them are compromised. To address the lack of a method to measure reliability, we propose a novel approach for estimating the reliability coefficient by extending the existing method for diagnostic classification models to time-series data. We apply the proposed method to actual response data and demonstrate its capability of assessing the reliability of BKT models, highlighting the importance of evaluating reliability and its potential for improving individualized learning experiences.