Self-Preference or Position Bias? A Controlled Multilingual Study of Evaluation Bias in LLM-as-Judge Pairwise Comparison
Background: Large language models (LLMs) are increasingly used as automated evaluators, but concerns remain about systematic biases such as self-preference.Prior findings may be confounded by stylistic similarity, shared quality preferences, or presentation order rather than genuine authorshipbased bias.Objective: We test whether LLM judges favor their own generated texts in blind pairwise comparison beyond what shared quality preferences would predict, and we quantify position bias as a competing explanation.Methods: We conduct a controlled, blind, multilingual experiment.Five near-frontier LLMs from five developers generate short texts in five European languages across three text types, with five generations per cell (375 texts).Each model then blindly compares its own text against each other model's text, in both presentation orders, yielding 3,000 evaluations.Self-selection is benchmarked against crossmodel selection of the same text as a quality-controlled baseline, and we apply equivalence testing to bound the size of small effects.Results: Across 3,000 evaluations, a correctly specified conditional-logit model (Bradley-Terry candidate strengths plus a self-bias term, controlling for both candidate quality and presentation order) revealed a modest but robust overall own-text preference (odds ratio or OR=1.26; p<.001).This was concentrated in two of the five judges (Gemini OR=2.09,Mistral OR=3.20; both p<.001); one judge showed the reverse (GPT OR=0.43; p<.001), and two showed no significant effect (Claude, Qwen).Model-specific position biases were strong (up to φ=0.62) and constitute the single largest distortion.Conclusion: Evaluation reveals a genuine own-text preference beyond shared quality, alongside strong position bias; both must be actively mitigated in LLM-as-judge pipelines.We do not attribute the owntext preference specifically to authorship recognition rather than familiarity, which our design does not separate.
Authors
- Tomáš Brückner (ORCID: https://orcid.org/0000-0001-9120-556X)
- Filip Vencovský (ORCID: https://orcid.org/0000-0002-4963-3912)
- Oleg Svatoš (ORCID: https://orcid.org/0000-0003-1955-9042)
Institutions
- Prague University of Economics and Business (CZ)
Publication Details
- Journal
- Acta Informatica Pragensia
- Published
- 2026-11-14
- DOI
- https://doi.org/10.18267/j.aip.336
- Primary Topic
- Ethics and Social Impacts of AI
- Type
- article
- Field-Weighted Citation Impact
- 0.00