Using peer review to evaluate the societal relevance of humanities research

Abstract In light of growing calls to demonstrate the societal relevance of academic work, this paper explores whether peer review can reliably evaluate the societal relevance of humanities research. It also estimates how relevant published journal articles and books from five humanities fields are to society. By modeling two evaluation tasks involving 38 early-career researchers and 885 humanities abstracts in English from Web of Science, we estimate how reviewer characteristics (such as their chauvinism and strictness) and document characteristics (such as field and content) affect societal relevance ratings. We then compare the influence of both reviewer and document characteristics on these ratings and estimate the societal relevance of humanities research where the factors contributing to peer review unreliability are controlled for. Although the study’s design and limited sample size necessitate cautious interpretation, the results of this study do provide tentative evidence that, even according to early-career researchers from the humanities, a substantial portion of published humanities research may not be relevant to society at large. Furthermore, these results also suggest that when using peer review to decide whether a particular piece of research is societally relevant, the selection of reviewers plays a more significant role than the content of the research.