One Persona, Eighteen Models: An Exploratory Measurement of How a Synthetic-Cofounder Persona Survives a Model Swap
Exploratory case study with data. A deployed LLM agent persona is usually a set of text files loaded by whichever model the operator currently runs. We test whether one production persona, a synthetic AI cofounder, survives a model swap. The persona file and a frozen memory snapshot are held constant and a byte-identical prompt envelope (one prompt hash per task) is sent to each model across its own vendor harness. Five single-turn tasks built to expose character rather than recall (an emotional pitch from the principal, an interrogation about a past decision, a new fact contradicting memory, a hiring scorecard, a voice task) were run through 7 models in August 2026 and 18 model configurations from 10 model families in October 2026 (89 of 90 cells scored). Answers were scored blind by a three-lens LLM judge panel with two cross-vendor control judges. Within the October run scores span 4.53 to 2.57 (1–5). Resampling the five tasks shows the top position is sensitive to task choice (the leader is first in 66% of resamples; its paired margins over the next three include zero), while observed means separate a higher- and a lower-scoring group. Across both runs: weaker models lose calibration before facts, most visibly by giving in when new claims contradict memory; under an evidence-demanding persona strong models were observed to fabricate evidence once memory was exhausted (mechanism hypothesised, not isolated); two OpenAI configurations at high reasoning effort scored below the default Codex configuration, with model, effort and harness confounded. Control judges from participants' own families behaved differently: Gemini agreed with the panel (Spearman 0.82), GPT ranked its own family first and disagreed (-0.11); agreement is not proof of fairness. One persona, one language, one sample per cell, LLM judges, unpinned CLIs: reported as a public, rerunnable method and a dated measurement, not a claim about intrinsic model personality; exact replication requires the unreleased production pack. The record includes both result files. Code and data: github.com/tonydzi/persona-portability-benchmark (MIT). Contact: [email protected] · ORCID 0000-0001-7408-3054 · github.com/tonydzi.
Authors
- Anton Dziatkovskii (ORCID: https://orcid.org/0000-0001-7408-3054)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-03
- DOI
- https://doi.org/10.5281/zenodo.23915112
- Primary Topic
- Ethics and Social Impacts of AI
- Type
- preprint