Psychometrics of an Elo-based large-scale online learning system

The Elo rating system (ERS), an intuitive and computationally efficient algorithm, offers a means to effectively update estimates of item difficulties and learner abilities as they evolve. This method proves to be highly advantageous in online learning environments. Computerized adaptive practice (CAP) endeavors to present learners with items that are well-suited to their individual ability levels, with the ultimate goal of enhancing motivation and optimizing learning outcomes. The objective of this paper is to outline common challenges that arise in an Elo-based CAP system and to present the psychometric enhancements implemented in the Prowise Learn environments to address these concerns. More specifically, we focus on three main aspects; 1) the development of a new scoring rule balancing response time and accuracy, 2) a way to fix the item scale to deal with item drift, and 3) an improved adaptive K-factor algorithm to speed up convergence in estimation. Using data from the Prowise Learn environment, analyses were done to illustrate the effect of the enhancements. Results show that these enhancements result in more dynamic tracking of the ratings, solve the issue of item drift, and capture the speed-accuracy trade-off more accurately.