An exploratory analysis of the operational restrictions of virtual peer review panels

Abstract In the early months of the COVID-19 pandemic, many research funding organizations were faced with the choice of suspending their peer review panels, or else continuing their decision-making processes virtually. Although seen part of a longer drive to make peer review more cost and time efficient as well as to combat climate and sustainability goals by reducing academic travel, it is still not fully understood how the adoption of the virtual peer review panels influences the decision-making process. Using an initial exploratory approach and using a series of observations of four peer review panels conducted virtually during 2020 at the Research Council of Norway (Forskningsrådet), this research explores how panellists behaved, and deliberations operated within virtual environments. The initial, exploratory findings show that despite arguments that virtual panels are more efficient, by saving time and money by allowing panellists to participate from their home settings, behaviours around the role of Panel Chair, and the ability of panellists to move in and out of deliberations, suggests alterations in how panels reached and then confirmed a consensus in the virtual environment. Deliberate mechanisms to confirm consensus were required during virtual panels, suggesting a more onerous workload mid- and post-panel for Panel Chairs and Call managers. Whereas a majority of panel groups had experience of working together as a panel in the past, the process of introducing new members in an online environment was problematic. These preliminary results indicate that more evidence is needed about how the virtual environment influences peer review processes before a more permanent change is adopted by funding agencies.