A Practical Guide to the Multi-Dimensional Elo Rating System

The multi-dimensional Elo rating system was designed to be a method of extending the more popular Elo system to model a wider variety of phenomena based on pairwise comparisons. This is useful in comparing the behaviours of machine learning models that would otherwise be black boxes. Unfortunately, the mElo rating system as it is currently formulated in the academic literature contains feedback loops that prevent the mElo rating system from correctly inferring an expected pairwise comparison matrix. In this paper, we propose an altered version of the mElo rating system that removes the feedback loops and ensures that a certain level of accuracy can be achieved. Furthermore, we demonstrate how the geometry associated with the model behind the multi-dimensional Elo rating system introduces complexities not present in the Elo rating system. We discuss some of these differences and how they affect the use and interpretation of the mElo rating system. We compare our version of the multi-dimensional Elo rating system to other rating systems and demonstrate that it provides the most precise estimates of an expected payoff matrix in realistic situations; we also discuss methods to help the reader select the learning rate. We conclude the paper with a list of recommendations for the practitioner to ensure that the mElo rating system is being used correctly to infer a unique pairwise comparison matrix.