Sceptics and champions: participant insights on the use of partial randomization to allocate research culture funding

Abstract As part of the shift towards a more equitable research culture, funders are reconsidering traditional approaches to peer review. In doing so, they seek to minimize bias towards certain research ideas and researcher profiles, to ensure greater inclusion of disadvantaged groups, to improve review quality, to reduce burden, and to enable more transformative research. This paper presents the rationale and findings from a trial of partially randomized funding allocation (PRA) used to internally distribute Research England’s Enhancing Research Culture fund at the University of Leeds. Of 26 eligible applications, six fell in the upper midfield on quality assessment and were randomized. Of this subset, one received the available funding. Qualitative data from applicants, reviewers, and moderators in the trial suggest modest gains regarding the reduction or perception of bias and efficiency of peer review. There were variable benefits of the feedback that PRA provides. A range of concerns emerged about PRA’s perceived fairness, bluntness, and threat to quality. Based on the behavioural responses we captured, we present strategic and operational recommendations for professionals working in research evaluation who are interested in adopting PRA as a way of improving research culture. For example, we highlight the importance of (1) sharing data on the role of chance in traditional peer review, and (2) considering the benefits and risks of PRA at both group and individual applicant levels.