The Archipelago of Thirty-Six: Measuring Structural Divergence in Large Language Model Judgment on Tasks Without a Correct Answer (Cross Model Repertory Grid)

Public leaderboards rank large language models (LLMs) by capability on tasks with a verifiable correct answer. A large class of practical tasks, including advisory, strategic, and ethical judgment, has no such answer: competent models can legitimately disagree. We introduce the Cross-Model Repertory Grid (CM-RG), an automated, contamination-resistant pipeline that adapts Kelly's Personal Construct Psychology to measure structural divergence in evaluative judgment between models rather than their capability. In the run reported here we covered 36 frontier models from 12 provider families across three deployment tiers (cheap, mid, flagship), on seven advisory tasks under two prompting conditions (neutral and persona). Models produced free responses, elicited their own bipolar constructs by triadic comparison, and cross-rated one another on the union of emergent constructs. The analysis of record (2026-06-13) comprises 3,055,153 ratings over 13,928 rater-by-ratee pairs and 86,418 emergent constructs, at a total API cost of $112.89. The mean pairwise inter-rater correlation was low. We report three structured endings: (1) weak consensus that survives every robustness check; (2) a calibration-tier eject in which a rater's own tier, not its target's, predicts how leniently it scores (mid-tier highest, cheap-tier lowest); and (3) a three-region structure, with a tight Western-lab cluster, a Chinese-flagship transitional zone, and an anti-consensus periphery of small open-weights models that correlate negatively with the consensus. We argue frontier LLM judgment on open tasks is pluralistic, structured, and largely orthogonal to capability tier and price. CM-RG is explicitly not appropriate for tasks with a verifiable correct answer.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-06-16
DOI
https://doi.org/10.5281/zenodo.20723228
Primary Topic
Computational and Text Analysis Methods
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

The Archipelago of Thirty-Six: Measuring Structural Divergence in Large Language Model Judgment on Tasks Without a Correct Answer (Cross Model Repertory Grid)

Sergey Dolgov
Zenodo (CERN European Organization for Nuclear Research)
Computational and Text Analysis Methods
preprint

The Archipelago of Thirty-Six: Measuring Structural Divergence in Large Language Model Judgment on Tasks Without a Correct Answer (Cross Model Repertory Grid)

Sergey Dolgov
preprint en

Abstract

Public leaderboards rank large language models (LLMs) by capability on tasks with a verifiable correct answer. A large class of practical tasks, including advisory, strategic, and ethical judgment, has no such answer: competent models can legitimately disagree. We introduce the Cross-Model Repertory Grid (CM-RG), an automated, contamination-resistant pipeline that adapts Kelly's Personal Construct Psychology to measure structural divergence in evaluative judgment between models rather than their capability. In the run reported here we covered 36 frontier models from 12 provider families across three deployment tiers (cheap, mid, flagship), on seven advisory tasks under two prompting conditions (neutral and persona). Models produced free responses, elicited their own bipolar constructs by triadic comparison, and cross-rated one another on the union of emergent constructs. The analysis of record (2026-06-13) comprises 3,055,153 ratings over 13,928 rater-by-ratee pairs and 86,418 emergent constructs, at a total API cost of $112.89. The mean pairwise inter-rater correlation was low. We report three structured endings: (1) weak consensus that survives every robustness check; (2) a calibration-tier eject in which a rater's own tier, not its target's, predicts how leniently it scores (mid-tier highest, cheap-tier lowest); and (3) a three-region structure, with a tight Western-lab cluster, a Chinese-flagship transitional zone, and an anti-consensus periphery of small open-weights models that correlate negatively with the consensus. We argue frontier LLM judgment on open tasks is pluralistic, structured, and largely orthogonal to capability tier and price. CM-RG is explicitly not appropriate for tasks with a verifiable correct answer.

Zenodo (CERN European Organization for Nuclear Research)
Peace, Justice and strong institutions
Computational and Text Analysis Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.