Estimation of partial rankings from sparse, noisy comparisons

Abstract Ranking items from pairwise comparisons is common in domains ranging from sports to consumer preferences. Statistical inference-based methods, such as the Bradley–Terry model, have emerged as flexible and powerful tools to tackle ranking in empirical data. However, in situations with limited and/or noisy comparisons, it is often challenging to confidently distinguish item performance based on evidence available in the data. Most ranking methods nevertheless force a complete ordering, suggesting a meaningful distinction when there is none. Here, we introduce a principled nonparametric Bayesian framework for learning partial rankings—rankings with ties—that infers distinctions between items only when supported by the evidence. We develop a fast agglomerative algorithm for Maximum a Posteriori (MAP) inference under this framework and evaluate its performance on a range of synthetic and real-world datasets, finding that it often yields a more parsimonious and reliable summary of the data than traditional ranking approaches, particularly in sparse observational settings.