Investigating the concept of consistency and an inconsistency index in two- and three-option stochastic paired comparison models

Abstract Paired comparison methods play a fundamental role in decision-making, preference elicitation, and data evaluation. While the concept of consistency is well established in pairwise comparison matrices-based models, the stochastic models, including Thurstone-motivated models, have remained less explored in this regard. This paper extends the notion of consistency in a groundbreaking way to two- and three-option Thurstone-motivated models, as well as Davidson’s model, establishing connections between the two-option Bradley-Terry, the three-option Davidson’s model, and pairwise comparison matrices-based models through equivalence statements. These statements are the basis for an alternative definition of consistency, valid for general two- and three-option stochastic paired comparison models. Based on this definition we also define a data inconsistency index to quantify deviations from ideal consistency because the consistent data sets are very rare. We formulated a set of axioms for data inconsistency indices and proved that they are independent and form a consistent axiomatic system. We also showed that the proposed inconsistency index satisfies all of these axioms. Based on random simulations, we illustrate its further properties. These results provide a unified framework for the assessment of consistency, paving the way for future developments of formal inconsistency indices.