National-background bias in scientific evaluation based on large language models
Abstract As large language models (LLMs) are increasingly used to gather, process and disseminate scientific information, concerns have grown about their identity-related biases. However, national-background biases in scientific evaluation remain insufficiently understood. Existing studies are limited to a narrow range of countries, abbreviated materials and/or simplified evaluation criteria, and have reported inconsistent findings. Moreover, differences in the textual evaluations generated by LLMs have seldom been quantified. Here, we used four popular LLMs to evaluate 292 full-text manuscripts from 28 journals across 8 research fields under 12 assigned national backgrounds, yielding 14,016 manuscript assessments, each covering 28 evaluative dimensions. Further comment-based rescoring tests were conducted to quantify differences in their textual evaluations. The results showed systematic differences in both scores and comments despite unchanged scientific content, with the magnitude and statistical significance these differences varying across national backgrounds. Under different national backgrounds, LLMs could offer different views on the same issue, and give different scores alongside similar textual evaluation. Researchers and decision-makers should be aware of these biases associated with source identity, including national background, when using LLMs for information gathering and processing.
Authors
- Hao Kong (ORCID: https://orcid.org/0000-0001-5038-2914)
- Wenjun Meng
- Chaoqun Ma
- Yafang Cheng
- Ruijing Ni
Institutions
- Max Planck Institute for Chemistry (DE)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-17
- DOI
- https://doi.org/10.21203/rs.3.rs-11755021/v1
- Primary Topic
- scientometrics and bibliometrics research
- Type
- preprint