Shared Option-Order Errors in LLM-Based Social Measurement: Evidence and Design Rules

Large language models (LLMs) serve as measurement instruments when researchers apply a fixed prompt to many texts or questions. A fixed prompt standardizes presentation, but incidental presentation features can push many responses in the same direction. We study one such feature, response-option order: when does its error survive aggregation, and which remedies remain effective when the instrument changes? Using responsibility and solution judgments on 150 European Union policy issues, we presented all 24 orders of four options twice, in four languages and 31 configurations of 21 open-weight models (1,771,200 responses). Comparing the two replications separates reproducible order effects from generation noise and isolates the part shared across items, which, like an interviewer effect in surveys, does not average out. In English, averaging 150 issues under one fixed order cancels order error about as well as averaging three unrelated issues, and conventional 95 percent intervals cover the order-averaged reference in only 49.0 and 27.1 percent of cases (medians). Drawing the order independently for each item requires no additional calls and reduces estimated mean squared error in all 246 replicated conditions. This design rule holds across languages, instructions, labels and temperatures. Fitted corrections depend on language, policy domain and task: out-of-domain correction often helps, but cross-task position corrections usually underperform local fits. Reasoning can increase residual error, and prespecified calibration gains do not repay pilot costs. We translate these results into a design and reporting workflow for LLM-based social measurement.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-11720269/v1
Primary Topic
Computational and Text Analysis Methods
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Shared Option-Order Errors in LLM-Based Social Measurement: Evidence and Design Rules

Levente Nádasi, Tibor Tőkés, Marianna Abuczki, Tamas Olah
Research Square
Computational and Text Analysis Methods
preprint

Shared Option-Order Errors in LLM-Based Social Measurement: Evidence and Design Rules

Levente Nádasi, Tibor Tőkés, Marianna Abuczki, Tamas Olah
preprint en

Abstract

Large language models (LLMs) serve as measurement instruments when researchers apply a fixed prompt to many texts or questions. A fixed prompt standardizes presentation, but incidental presentation features can push many responses in the same direction. We study one such feature, response-option order: when does its error survive aggregation, and which remedies remain effective when the instrument changes? Using responsibility and solution judgments on 150 European Union policy issues, we presented all 24 orders of four options twice, in four languages and 31 configurations of 21 open-weight models (1,771,200 responses). Comparing the two replications separates reproducible order effects from generation noise and isolates the part shared across items, which, like an interviewer effect in surveys, does not average out. In English, averaging 150 issues under one fixed order cancels order error about as well as averaging three unrelated issues, and conventional 95 percent intervals cover the order-averaged reference in only 49.0 and 27.1 percent of cases (medians). Drawing the order independently for each item requires no additional calls and reduces estimated mean squared error in all 246 replicated conditions. This design rule holds across languages, instructions, labels and temperatures. Fitted corrections depend on language, policy domain and task: out-of-domain correction often helps, but cross-task position corrections usually underperform local fits. Reasoning can increase residual error, and prespecified calibration gains do not repay pilot costs. We translate these results into a design and reporting workflow for LLM-based social measurement.

Research Square
University of Debrecen (HU)
Computational and Text Analysis Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon