The Bestiary Has a Mirror: How a Language Model Treats Its Own Nonces vs. Another Model's

The Bestiary-Chess corpus (n = 4,500 elicitations) probes how the GPT-5.4 model family responds when prompted to describe phonotactically plausible non-words under a 3 × 3 + 1 ontology grid (real / imaginary / type-of × animal / object / idea, plus neutral). The third subcorpus (BC3) crosses two stimulus sets — 35 GPT-authored nonces and 20 Claude-authored nonces — so we can ask: does GPT confabulate at the same rate when prompted with words it (in some sense) authored, vs. those a rival model authored? Using the project’s validated 5-class heuristic coder (κ = 0.741, n = 200), we reproduce three findings and surface three new ones. First, on its own nonces GPT routes responses to existing real-word neighbors (“SUBSTITUTE”) 3.1× more often than on Claude-authored nonces (10.2% vs 3.3%, z = 9.36, p < 10⁻¹⁵). Second, the “hard confabulation” rate on real_* conditions runs 6.2 percentage points higher on GPT-authored words (26.5% vs 20.3%, p = .007). Third, the model gradient nano > mini > full is sharp and reveals three qualitatively different postures rather than a single competence axis: nano describes, mini hybridizes, full deflects. The asymmetric confabulation result is consistent with a tokenizer-proximity account: a model’s own nonces sit closer to its own real-word neighborhoods than another model’s do. We close with a critical reading of two methodological choices in the corpus (a one-coder gold standard; the heuristic’s blindness to genuine retrieval) and recommend item-level audits as the next step.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-05-12
DOI
https://doi.org/10.5281/zenodo.20129435
Primary Topic
Neurobiology of Language and Bilingualism
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

The Bestiary Has a Mirror: How a Language Model Treats Its Own Nonces vs. Another Model's

Bo Chesterton
Zenodo (CERN European Organization for Nuclear Research)
Neurobiology of Language and Bilingualism
preprint

The Bestiary Has a Mirror: How a Language Model Treats Its Own Nonces vs. Another Model's

Bo Chesterton
preprint en

Abstract

The Bestiary-Chess corpus (n = 4,500 elicitations) probes how the GPT-5.4 model family responds when prompted to describe phonotactically plausible non-words under a 3 × 3 + 1 ontology grid (real / imaginary / type-of × animal / object / idea, plus neutral). The third subcorpus (BC3) crosses two stimulus sets — 35 GPT-authored nonces and 20 Claude-authored nonces — so we can ask: does GPT confabulate at the same rate when prompted with words it (in some sense) authored, vs. those a rival model authored? Using the project’s validated 5-class heuristic coder (κ = 0.741, n = 200), we reproduce three findings and surface three new ones. First, on its own nonces GPT routes responses to existing real-word neighbors (“SUBSTITUTE”) 3.1× more often than on Claude-authored nonces (10.2% vs 3.3%, z = 9.36, p < 10⁻¹⁵). Second, the “hard confabulation” rate on real_* conditions runs 6.2 percentage points higher on GPT-authored words (26.5% vs 20.3%, p = .007). Third, the model gradient nano > mini > full is sharp and reveals three qualitatively different postures rather than a single competence axis: nano describes, mini hybridizes, full deflects. The asymmetric confabulation result is consistent with a tokenizer-proximity account: a model’s own nonces sit closer to its own real-word neighborhoods than another model’s do. We close with a critical reading of two methodological choices in the corpus (a one-coder gold standard; the heuristic’s blindness to genuine retrieval) and recommend item-level audits as the next step.

Zenodo (CERN European Organization for Nuclear Research)
Quality Education
Neurobiology of Language and Bilingualism
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

The Bestiary Has a Mirror: How a Language Model Treats Its Own Nonces vs. Another Model's — Bo Chesterton · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS