A supervised machine learning approach for assessing grant peer review reports

Abstract Peer review is essential to the research lifecycle, yet the contents of grant peer review reports remain underexplored. Our study addresses this gap by developing a pipeline to systematically analyze these reports using natural language processing and machine learning. We define 12 categories relevant to funding agencies, create an annotation codebook, fine-tune and validate transformer models, and apply these classifiers to a novel text corpus consisting of 1.6 million sentences from 47,522 grant peer review reports submitted to the Swiss National Science Foundation. This work has critical implications for the academic community. It provides novel insights into the content of grant peer review reports and openly available tools to enhance transparency, fairness, and consistency in grant evaluation. Our findings also highlight differences between journal and grant peer reviews, while the developed framework enables funding agencies and researchers to refine practices, fostering a more trustworthy and efficient evaluation process.