Toward Psychometric Learning Analytics: Augmenting the Urnings Algorithm with Response Times

Abstract Adaptive learning systems (ALS) aim to tailor the educational material to match the student's needs, ultimately improving the learning outcomes. An ALS dynamically adjust the level of practice based on the student's ability; therefore, obtaining accurate ability estimates is crucial. Since the amount of responses in a timeframe is limited, high measurement precision is unattainable using only accuracies, which calls for the inclusion of other data sources into the measurement. Here, we propose algorithms that can estimate the abilities on‐the‐fly based on both accuracy and response times (RT). These are extensions of a rating system called the Urnings algorithm. Since the Urnings algorithm uses discrete updates, building on the difference between a single observed and simulated response, we combined accuracy and RT into a continuous score using a discretized version of the Signed Residual Time (SRT) scoring rule. Through simulation studies, we showed that by augmenting the algorithm with RT, a reliable ability measure and better ability tracking can be obtained by administering fewer items. By reanalyzing data from an existing ALS, we showed that the algorithms can be utilized even if the SRT scoring rule is not explicitly used during measurement, providing better ability estimates and smaller prediction errors.