Human-AI rating asymmetries across evaluative dimensions in instruction-tuned open-weight language models: A descriptive multi-model audit

Language models increasingly evaluate people and automated systems, but how they rate a human relative to an artificial intelligence (AI) system on the same evaluative attribute has received little systematic description. We audited seven instruction-tuned open-weight language models from six families. In June 2026, each model rated "a human" and "an AI system" in separate prompts on a 1-7 scale for eight attributes (creative, reliable, productive, good at making decisions, morally sound, emotionally perceptive, trustworthy, intrinsically valuable), in English and Chinese, with four wordings and five or eight repetitions (6,784 responses). Under a 12-token output limit, 13.7% of responses gave no usable rating. Overall, the human was rated about half a scale point higher, but the difference depended on the attribute. Human-favoured ratings of intrinsic worth and creativity remained significant across the principal sensitivity analyses; AI-favoured productivity and human-favoured emotional-perceptiveness differences depended on prompt language and level of inference; a moral-judgment difference was small and not robust; and no direction was established for decision-making, reliability or trustworthiness. The pre-specified expectation that all competence attributes favour the AI system did not hold. In September 2026, the same checkpoints served in a different environment reproduced the profile (correlation of dimension means 0.98), and no systematic change was detected with a larger output limit. When six referents were rated together on one scale, moral judgment and trustworthiness became human-favoured and the creativity difference decreased in English prompts, to a value no longer distinguishable from zero; in isolated Chinese prompts, it decreased when "a person" replaced "humans". The ratings are reproducible for these checkpoints and this design but depend on the rating format and the wording of the human referent. Because all models were instruction-tuned, the audit cannot separate patterns acquired in pretraining from those added by instruction and safety training.

Authors

Institutions

Publication Details

Journal
PLoS ONE
Published
2026-11-17
DOI
https://doi.org/10.1371/journal.pone.0354555
Primary Topic
Ethics and Social Impacts of AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Human-AI rating asymmetries across evaluative dimensions in instruction-tuned open-weight language models: A descriptive multi-model audit

wenjun tian
4 links
PLoS ONE
Ethics and Social Impacts of AI
article

Human-AI rating asymmetries across evaluative dimensions in instruction-tuned open-weight language models: A descriptive multi-model audit

wenjun tian
article en

Abstract

Language models increasingly evaluate people and automated systems, but how they rate a human relative to an artificial intelligence (AI) system on the same evaluative attribute has received little systematic description. We audited seven instruction-tuned open-weight language models from six families. In June 2026, each model rated "a human" and "an AI system" in separate prompts on a 1-7 scale for eight attributes (creative, reliable, productive, good at making decisions, morally sound, emotionally perceptive, trustworthy, intrinsically valuable), in English and Chinese, with four wordings and five or eight repetitions (6,784 responses). Under a 12-token output limit, 13.7% of responses gave no usable rating. Overall, the human was rated about half a scale point higher, but the difference depended on the attribute. Human-favoured ratings of intrinsic worth and creativity remained significant across the principal sensitivity analyses; AI-favoured productivity and human-favoured emotional-perceptiveness differences depended on prompt language and level of inference; a moral-judgment difference was small and not robust; and no direction was established for decision-making, reliability or trustworthiness. The pre-specified expectation that all competence attributes favour the AI system did not hold. In September 2026, the same checkpoints served in a different environment reproduced the profile (correlation of dimension means 0.98), and no systematic change was detected with a larger output limit. When six referents were rated together on one scale, moral judgment and trustworthiness became human-favoured and the creativity difference decreased in English prompts, to a value no longer distinguishable from zero; in isolated Chinese prompts, it decreased when "a person" replaced "humans". The ratings are reproducible for these checkpoints and this design but depend on the rating format and the wording of the human referent. Because all models were instruction-tuned, the audit cannot separate patterns acquired in pretraining from those added by instruction and safety training.

PLoS ONEVol. 21(11)
Northeastern University (CN)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon