An observational study of the Patient-Centered Outcomes Research Institute’s switch from in-person to virtual research application review panels during the COVID-19 pandemic

BACKGROUND: The Patient-Centered Outcomes Research Institute (PCORI) reviews research applications using a process that includes preliminary review by primary reviewers, followed by panel discussion where primary reviewers and other reviewers on the panel (panelists) discuss and score the most competitive applications. PCORI switched from in-person to virtual panel discussions during the COVID-19 pandemic. This study compared reviewers' scores and perceptions of panel interactions for virtual and in-person review panels and assessed reviewers' perspectives about virtual panels. PCORI uniquely includes patients and other stakeholders as reviewers alongside scientists; thus, differences between reviewer types were also explored. METHODS: This observational study utilized repeated cross-sectional data from PCORI funding cycles before and after switching to virtual review, including reviewer score data and closed- and open-ended responses from reviewer surveys. The study team used linear and logistic regression to examine reviewer scores and quantitative survey responses and conducted thematic analysis of open-ended survey responses. RESULTS: There were no statistically significant differences between in-person and virtual panels in primary reviewers' post-discussion scores or the magnitude of score change from pre- to post-discussion. In contrast, panelists gave stronger post-discussion scores in virtual panels, and this difference between formats was greater for stakeholder reviewers compared to scientists. Reviewers' ratings of panel interactions appeared to be similar between formats. While reviewers noted challenges of virtual panels, including challenges related to discussion quality and social interactions, they also noted benefits, such as lack of travel. CONCLUSIONS: Overall, this study suggests that virtual and in-person review panels were largely comparable on reviewer scores and some key aspects of reviewer experiences in a multi-stakeholder review process. Despite having some challenges, virtual review processes may be a viable approach for funders during times of necessity and beyond. Practical considerations and recommendations for virtual panels are discussed.