Judgement Noise in a Complex Evaluation System: Why the UK Research Excellence Framework Needs a Noise Audit

Expert judgement is inherently noisy, but the consequences of this variability are often underestimated. Here I examine the UK Research Excellence Framework (REF) as a case study illustrating how judgment noise can interact with structural features of a multi-stage evaluation system. Simulations show that noise in both internal departmental reviews and REF panel reviews, in conjunction with the bounded REF rating scale, produces systematic underestimation of the highest-performing departments. Panel noise, rather than internal noise, had a disproportionate impact on department rankings, with even moderate noise producing severe distortions. At a policy level, these results underscore the need for a noise audit of REF panel ratings. More broadly, the case of REF demonstrates that noise embedded in sequential evaluation, for example, in hiring and grant review decisions, does not “cancel out”, but instead can propagate in potentially surprising ways.