Parameter Estimation in Comparative Judgment Under Random and Adaptive Scheduling Schemes

Abstract Comparative judgment is an assessment method where item ratings are estimated based on rankings of subsets of the items. These rankings are typically pairwise, with ratings taken to be the estimated parameters from fitting a Bradley‐Terry model. Likelihood penalization is often employed to ensure finiteness of estimates. Adaptive scheduling of the comparisons can increase the efficiency of the assessment. We show that the most commonly used penalty in Comparative Judgment is not the best‐performing penalty under adaptive scheduling and can lead to substantial bias in parameter estimation. We demonstrate this using simulated and real data and provide a theoretical explanation for the relative performance of the penalties considered, including identifying a preferred alternative. Further, we propose a novel approach based on a parametric bootstrap. It is found to produce better parameter estimates for adaptive schedules and to be robust to variations in underlying strength distributions. The work allows for more efficient implementations of comparative judgment.