How Model Choice and Memory Shape Preference Consistency in Large Language Models

When a language model answers a sequence of questions, researchers report the model and temperature but rarely what or how it was told about its own earlier answers. I show that this underreported setting behaves as a mode effect. Testing four large language model (LLM) families with revealed preference theory (GARP) across moral, economic, and social tasks ( N = 5,926 ), I find that how prior choices re-enter the prompt, the memory condition , moves the share of internally consistent subjects by up to 48 percentage points, mostly where that share starts lowest. Memory also influences which preferences the models express and, in a silicon-sampling extension, it brings the answers closer to the human distribution. Nonetheless, gains in consistency and surface fidelity do not necessarily translate into representativeness: a consistent, human-looking model may not represent any human population. For LLMs as social-science instruments, this study offers promise, caution, and a method for testing the stability of sequential outputs.

Authors

Institutions

Publication Details

Journal
Sociological Methods & Research
Published
2026-09-29
DOI
https://doi.org/10.1177/00491241261492203
Primary Topic
Language and cultural evolution
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
article

How Model Choice and Memory Shape Preference Consistency in Large Language Models

Yick Chung
Sociological Methods & Research
Language and cultural evolution
article

How Model Choice and Memory Shape Preference Consistency in Large Language Models

Yick Chung
article en

Abstract

When a language model answers a sequence of questions, researchers report the model and temperature but rarely what or how it was told about its own earlier answers. I show that this underreported setting behaves as a mode effect. Testing four large language model (LLM) families with revealed preference theory (GARP) across moral, economic, and social tasks ( N = 5,926 ), I find that how prior choices re-enter the prompt, the memory condition , moves the share of internally consistent subjects by up to 48 percentage points, mostly where that share starts lowest. Memory also influences which preferences the models express and, in a silicon-sampling extension, it brings the answers closer to the human distribution. Nonetheless, gains in consistency and surface fidelity do not necessarily translate into representativeness: a consistent, human-looking model may not represent any human population. For LLMs as social-science instruments, this study offers promise, caution, and a method for testing the stability of sequential outputs.

Sociological Methods & Research
University of Milan (IT)
Reduced inequalities
Openalex Percentile: Top 2%
Language and cultural evolution
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.