Benchmarking Choice Consistency in Large Language Models: A Measurement-Theoretic Analysis of Transitivity Axiom

Abstract While large language models (LLMs) are increasingly deployed as decision-makers, current evaluation practice emphasizes benchmark accuracy rather than the structural properties that make decisions coherent and interpretable. We introduce a measurement-theoretic framework for evaluating whether the choices of an LLM are transitively rationalizable, focusing on the choice axiom of transitivity of preference. When transitivity is violated over a choice set, alternatives can no longer be coherently ordered as globally better or worse, and a stated preference between one pair constrains nothing about how the model would choose between any others. We apply our framework across 20 distinct LLMs, systematically varying question format, generation temperature, and contextual memory, using classic choice paradigms previously shown to induce intransitive preferences in humans. We find substantial and systematic transitivity violations in many models, although some patterns partially mirror known features of human decision behavior. Testing the transitivity axiom thus offers a principled, complementary foundation for evaluating the consistency of AI decision making beyond benchmark accuracy.

Authors

Institutions

Publication Details

Journal
Computational Brain & Behavior
Published
2026-11-17
DOI
https://doi.org/10.1007/s42113-026-00342-3
Citations
1
Primary Topic
Decision-Making and Behavioral Economics
Type
article
Field-Weighted Citation Impact
2.80

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Benchmarking Choice Consistency in Large Language Models: A Measurement-Theoretic Analysis of Transitivity Axiom

Clintin P. Davis‐Stober, Kiwon Song, James M. Jennings
1 citations
Computational Brain & Behavior
Decision-Making and Behavioral Economics
2.80
article

Benchmarking Choice Consistency in Large Language Models: A Measurement-Theoretic Analysis of Transitivity Axiom

Clintin P. Davis‐Stober, Kiwon Song, James M. Jennings
article en
1 citations

Abstract

Abstract While large language models (LLMs) are increasingly deployed as decision-makers, current evaluation practice emphasizes benchmark accuracy rather than the structural properties that make decisions coherent and interpretable. We introduce a measurement-theoretic framework for evaluating whether the choices of an LLM are transitively rationalizable, focusing on the choice axiom of transitivity of preference. When transitivity is violated over a choice set, alternatives can no longer be coherently ordered as globally better or worse, and a stated preference between one pair constrains nothing about how the model would choose between any others. We apply our framework across 20 distinct LLMs, systematically varying question format, generation temperature, and contextual memory, using classic choice paradigms previously shown to induce intransitive preferences in humans. We find substantial and systematic transitivity violations in many models, although some patterns partially mirror known features of human decision behavior. Testing the transitivity axiom thus offers a principled, complementary foundation for evaluating the consistency of AI decision making beyond benchmark accuracy.

Computational Brain & Behavior
Oregon State University (US), University of Missouri (US)
National Science Foundation, University of Missouri, National Institute on Alcohol Abuse and Alcoholism
Openalex Percentile: Top 18%
Decision-Making and Behavioral Economics
2.80
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon