Large language models and responsible research evaluation: an extension of the Leiden Manifesto

Abstract Research evaluators and scientometricians sometimes promote the message of responsible bibliometrics through initiatives like the Leiden Manifesto, but these do not mention Large Language Models (LLMs). Since there is evidence that LLMs can make quality predictions for journal articles that correlate more strongly with expert judgements than do citation-based indicators in most fields, they may start to supplement or replace citation-based indicators for some applications. This paper compares responsible evaluation principles from the Leiden Manifesto with those necessary for LLMs, finding them all to be still relevant. It also includes a discussion of how they apply to human expert review and the differences between the three approaches. For example, transparency is inherently weak for LLMs, since their decision-making processes are too complex to fully understand (similarly for experts). Conversely, LLMs may be able to address some issues that citation-based indicators can’t, such as by adapting prompts to the goals of an evaluation. Other issues impact differently. For example, LLM evaluations might encourage authors to customize articles to persuade LLMs, whereas bibliometric evaluations might encourage authors to solicit citations. Finally, two additions to the Leiden Manifesto targeting LLM-supported research evaluations are proposed. First, the cost/benefit tradeoff should be considered when deciding which approach to use. In resource-poor contexts or for minor evaluations, it may be irresponsible to implement otherwise fully responsible solutions. Second, LLM evaluations need to comply with national copyright law when processing academic texts: the ability to access a text does not always entail permission to automatically process it.