Self-Preference or Position Bias? A Controlled Multilingual Study of Evaluation Bias in LLM-as-Judge Pairwise Comparison

Background: Large language models (LLMs) are increasingly used as automated evaluators, but concerns remain about systematic biases such as self-preference.Prior findings may be confounded by stylistic similarity, shared quality preferences, or presentation order rather than genuine authorshipbased bias.Objective: We test whether LLM judges favor their own generated texts in blind pairwise comparison beyond what shared quality preferences would predict, and we quantify position bias as a competing explanation.Methods: We conduct a controlled, blind, multilingual experiment.Five near-frontier LLMs from five developers generate short texts in five European languages across three text types, with five generations per cell (375 texts).Each model then blindly compares its own text against each other model's text, in both presentation orders, yielding 3,000 evaluations.Self-selection is benchmarked against crossmodel selection of the same text as a quality-controlled baseline, and we apply equivalence testing to bound the size of small effects.Results: Across 3,000 evaluations, a correctly specified conditional-logit model (Bradley-Terry candidate strengths plus a self-bias term, controlling for both candidate quality and presentation order) revealed a modest but robust overall own-text preference (odds ratio or OR=1.26; p<.001).This was concentrated in two of the five judges (Gemini OR=2.09,Mistral OR=3.20; both p<.001); one judge showed the reverse (GPT OR=0.43; p<.001), and two showed no significant effect (Claude, Qwen).Model-specific position biases were strong (up to φ=0.62) and constitute the single largest distortion.Conclusion: Evaluation reveals a genuine own-text preference beyond shared quality, alongside strong position bias; both must be actively mitigated in LLM-as-judge pipelines.We do not attribute the owntext preference specifically to authorship recognition rather than familiarity, which our design does not separate.

Authors

Institutions

Publication Details

Journal
Acta Informatica Pragensia
Published
2026-11-14
DOI
https://doi.org/10.18267/j.aip.336
Primary Topic
Ethics and Social Impacts of AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Self-Preference or Position Bias? A Controlled Multilingual Study of Evaluation Bias in LLM-as-Judge Pairwise Comparison

Tomáš Brückner, Filip Vencovský, Oleg Svatoš
Acta Informatica Pragensia
Ethics and Social Impacts of AI
article

Self-Preference or Position Bias? A Controlled Multilingual Study of Evaluation Bias in LLM-as-Judge Pairwise Comparison

Tomáš Brückner, Filip Vencovský, Oleg Svatoš
article en

Abstract

Background: Large language models (LLMs) are increasingly used as automated evaluators, but concerns remain about systematic biases such as self-preference.Prior findings may be confounded by stylistic similarity, shared quality preferences, or presentation order rather than genuine authorshipbased bias.Objective: We test whether LLM judges favor their own generated texts in blind pairwise comparison beyond what shared quality preferences would predict, and we quantify position bias as a competing explanation.Methods: We conduct a controlled, blind, multilingual experiment.Five near-frontier LLMs from five developers generate short texts in five European languages across three text types, with five generations per cell (375 texts).Each model then blindly compares its own text against each other model's text, in both presentation orders, yielding 3,000 evaluations.Self-selection is benchmarked against crossmodel selection of the same text as a quality-controlled baseline, and we apply equivalence testing to bound the size of small effects.Results: Across 3,000 evaluations, a correctly specified conditional-logit model (Bradley-Terry candidate strengths plus a self-bias term, controlling for both candidate quality and presentation order) revealed a modest but robust overall own-text preference (odds ratio or OR=1.26; p<.001).This was concentrated in two of the five judges (Gemini OR=2.09,Mistral OR=3.20; both p<.001); one judge showed the reverse (GPT OR=0.43; p<.001), and two showed no significant effect (Claude, Qwen).Model-specific position biases were strong (up to φ=0.62) and constitute the single largest distortion.Conclusion: Evaluation reveals a genuine own-text preference beyond shared quality, alongside strong position bias; both must be actively mitigated in LLM-as-judge pipelines.We do not attribute the owntext preference specifically to authorship recognition rather than familiarity, which our design does not separate.

Acta Informatica Pragensia
Prague University of Economics and Business (CZ)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon