The use of expert judgement and pairwise comparison in the establishment of an assessment scale

This paper explores the validation of an assessment by two complementary processes. Expert judgment assigns difficulty values to items on the basis of long-standing knowledge of as assessment scale; pairwise comparison establishes item difficulty values through statistical analysis. Given that expert judgement is accepted as the cornerstone of the accurate setting of test items at particular points on a proficiency scale, the accuracy of the expert judgement is crucial. In the study, the statistical pairwise comparison of test items provides a Rasch-calibrated, external perspective on item difficulty. Twelve experienced item writers rated a set of 23 multiple-choice items drawn from a large set of calibrated items via pairwise comparison principles. In the analysis, good fit statistics emerged, with difficulty values very comparable to the original expert-proposed values. The study validates how ‘expert’ item writers can be seen to produce accurately-located items, and that faith may be placed in their judgements.