How Model Choice and Memory Shape Preference Consistency in Large Language Models
When a language model answers a sequence of questions, researchers report the model and temperature but rarely what or how it was told about its own earlier answers. I show that this underreported setting behaves as a mode effect. Testing four large language model (LLM) families with revealed preference theory (GARP) across moral, economic, and social tasks ( N = 5,926 ), I find that how prior choices re-enter the prompt, the memory condition , moves the share of internally consistent subjects by up to 48 percentage points, mostly where that share starts lowest. Memory also influences which preferences the models express and, in a silicon-sampling extension, it brings the answers closer to the human distribution. Nonetheless, gains in consistency and surface fidelity do not necessarily translate into representativeness: a consistent, human-looking model may not represent any human population. For LLMs as social-science instruments, this study offers promise, caution, and a method for testing the stability of sequential outputs.
Authors
- Yick Chung (ORCID: https://orcid.org/0009-0007-9286-279X)
Institutions
- University of Milan (IT)
Publication Details
- Journal
- Sociological Methods & Research
- Published
- 2026-09-29
- DOI
- https://doi.org/10.1177/00491241261492203
- Primary Topic
- Language and cultural evolution
- Type
- article
- Field-Weighted Citation Impact
- 0.00