The Bestiary Has a Mirror: How a Language Model Treats Its Own Nonces vs. Another Model's
The Bestiary-Chess corpus (n = 4,500 elicitations) probes how the GPT-5.4 model family responds when prompted to describe phonotactically plausible non-words under a 3 × 3 + 1 ontology grid (real / imaginary / type-of × animal / object / idea, plus neutral). The third subcorpus (BC3) crosses two stimulus sets — 35 GPT-authored nonces and 20 Claude-authored nonces — so we can ask: does GPT confabulate at the same rate when prompted with words it (in some sense) authored, vs. those a rival model authored? Using the project’s validated 5-class heuristic coder (κ = 0.741, n = 200), we reproduce three findings and surface three new ones. First, on its own nonces GPT routes responses to existing real-word neighbors (“SUBSTITUTE”) 3.1× more often than on Claude-authored nonces (10.2% vs 3.3%, z = 9.36, p < 10⁻¹⁵). Second, the “hard confabulation” rate on real_* conditions runs 6.2 percentage points higher on GPT-authored words (26.5% vs 20.3%, p = .007). Third, the model gradient nano > mini > full is sharp and reveals three qualitatively different postures rather than a single competence axis: nano describes, mini hybridizes, full deflects. The asymmetric confabulation result is consistent with a tokenizer-proximity account: a model’s own nonces sit closer to its own real-word neighborhoods than another model’s do. We close with a critical reading of two methodological choices in the corpus (a one-coder gold standard; the heuristic’s blindness to genuine retrieval) and recommend item-level audits as the next step.
Authors
- Bo Chesterton (ORCID: https://orcid.org/0009-0005-1312-6432)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-05-12
- DOI
- https://doi.org/10.5281/zenodo.20129435
- Primary Topic
- Neurobiology of Language and Bilingualism
- Type
- preprint