Pairwise-Elo rating system

Abstract This paper proposes a binary time-series model which captures intransitivity in paired comparison by extending the classic Elo Rating System (ELO). Most existing rating systems assume that players’ ratings are totally ordered and that transitivity holds, which precludes situations where a player performs exceptionally well against a specific opponent regardless of their overall ability. The proposed model, called the Pairwise-Elo (P-ELO) model, allows us to (i) construct a hypothesis test for the existence of pairwise advantages (i.e., intransitivity) across the players, and (ii) estimate winning probabilities for each pair while incorporating these pairwise effects. We compare P-ELO to ELO and mElo 2 k on Sumo Wrestling, Mixed Martial Arts, StarCraft II, and Large Language Models (LLMs) evaluations, in terms of match outcome prediction and probability estimation.