Wine Stars & Bars: The combinatorics of critic consensus and the usefulness of order preference models

Abstract Sums of the ratings that judges assign to wines are a near universal method of determining the winners and losers of wine competitions. Sums are easy to calculate and easy to communicate, but seven flaws make sums of ratings a perilous guide to relative quality or preference. Stars & Bars combinatorics show that the same sum can be the result of billions of compositions of ratings and that those compositions, for the same sum, can contain dispersion that ranges from universal consensus to apparent randomness to polar disagreement. Order preference models can address both order and dispersion, and an example using a Plackett–Luce model yields maximum likelihood estimates of top-choice probabilities that are a defensible guide to relative quality or preference.