Can generative AI effectively perform quality evaluation within social sciences? A case study in library and information science

Abstract Thus far, the usage of generative artificial intelligence (GAI) has mainly been explored for content-based evaluations. Research quality evaluations and studies focusing on fields in the social sciences are needed. Therefore, we performed a quality evaluation using state-of-the-art GAI models to assess their usability in this respect. GPT-4o and DeepSeek-V3 were employed to select papers with very high quality from a set of high quality papers. Comparisons with human expert decisions, citation counts, and download counts were the basis for our quality assessments. GAI models can to some extent assess the academic value of high-quality papers. We found a positive but limited correlation between GAI-based quality scores with citation and download counts. We observed selective preferences of GAI’s quality assessment in favor of papers that use overt technical terminology, methodological frameworks, or seemingly broad but objectively worded expressions. GAI comparatively undervalues the quality of papers with high theoretical abstraction or strong socio-contextual embedding. Our study provides new empirical foundations for defining GAI’s role in research quality evaluation in the social sciences. Crucial guidance for academic institutions, libraries, funding agencies, and research management organizations is extracted. We expect that GAI trained to assess academic quality would produce classic paper selections in higher agreement with human experts than currently available models.