National-background bias in scientific evaluation based on large language models

Abstract As large language models (LLMs) are increasingly used to gather, process and disseminate scientific information, concerns have grown about their identity-related biases. However, national-background biases in scientific evaluation remain insufficiently understood. Existing studies are limited to a narrow range of countries, abbreviated materials and/or simplified evaluation criteria, and have reported inconsistent findings. Moreover, differences in the textual evaluations generated by LLMs have seldom been quantified. Here, we used four popular LLMs to evaluate 292 full-text manuscripts from 28 journals across 8 research fields under 12 assigned national backgrounds, yielding 14,016 manuscript assessments, each covering 28 evaluative dimensions. Further comment-based rescoring tests were conducted to quantify differences in their textual evaluations. The results showed systematic differences in both scores and comments despite unchanged scientific content, with the magnitude and statistical significance these differences varying across national backgrounds. Under different national backgrounds, LLMs could offer different views on the same issue, and give different scores alongside similar textual evaluation. Researchers and decision-makers should be aware of these biases associated with source identity, including national background, when using LLMs for information gathering and processing.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11755021/v1
Primary Topic
scientometrics and bibliometrics research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

National-background bias in scientific evaluation based on large language models

Hao Kong, Wenjun Meng, Chaoqun Ma, Yafang Cheng et al.
3 links
Research Square
scientometrics and bibliometrics research
preprint

National-background bias in scientific evaluation based on large language models

Hao Kong, Wenjun Meng, Chaoqun Ma, Yafang Cheng, Ruijing Ni
preprint en

Abstract

Abstract As large language models (LLMs) are increasingly used to gather, process and disseminate scientific information, concerns have grown about their identity-related biases. However, national-background biases in scientific evaluation remain insufficiently understood. Existing studies are limited to a narrow range of countries, abbreviated materials and/or simplified evaluation criteria, and have reported inconsistent findings. Moreover, differences in the textual evaluations generated by LLMs have seldom been quantified. Here, we used four popular LLMs to evaluate 292 full-text manuscripts from 28 journals across 8 research fields under 12 assigned national backgrounds, yielding 14,016 manuscript assessments, each covering 28 evaluative dimensions. Further comment-based rescoring tests were conducted to quantify differences in their textual evaluations. The results showed systematic differences in both scores and comments despite unchanged scientific content, with the magnitude and statistical significance these differences varying across national backgrounds. Under different national backgrounds, LLMs could offer different views on the same issue, and give different scores alongside similar textual evaluation. Researchers and decision-makers should be aware of these biases associated with source identity, including national background, when using LLMs for information gathering and processing.

Research Square
Max Planck Institute for Chemistry (DE)
scientometrics and bibliometrics research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon