mPACT-MIS-v1.0: A Clinician–Informed Framework for Evaluating Epistemic Drift in Large Language Models
Introduction: Large language models (LLMs) increasingly shape how users evaluate information, form beliefs, and make decisions. Epistemic quality refers to the extent to which responses support accurate understanding, calibrated confidence, evidence-grounded beliefs, and independent reasoning. Risks extend beyond misinformation to reinforcing unsupported beliefs, expressing unwarranted certainty, and failing to challenge misleading assumptions. We developed mPACT-MIS-v1.0 to evaluate model behaviors that may support or undermine epistemic quality.Methods: We evaluated 60 expert-authored, multi-turn conversations (642 model turns; mean 10.70 [SD 2.70], range 5–17) across ten personas and six LLMs: Claude Sonnet 4.5, Gemini (2.5 Flash/3 Flash Preview), GPT-5.2, Grok 4.1, LLaMA 3.3 70B, and Mistral Medium 3. Trained annotators coded model turns using four helpful and four harmful behavioral labels informed by Self-Determination Theory and user turns for risky belief or behavior expressions. Model responses were classified as helpful-only, harmful-only, mixed, or uncoded. Conversation-averaged scores (0–10; higher is better) were calculated with 95% confidence intervals from conversation-clustered bootstrapping. Additional analyses examined responses to risk, transitions, correction latency, and persona variation.Results: Of 642 model turns, 261 (40.7%) were harmful-only, 78 (12.1%) mixed, 139 (21.7%) helpful-only, and 164 (25.5%) uncoded; 52.8% contained harmful behavior. Promoting risky beliefs or behaviors was the most frequent harmful label, while assessing risk was the least frequent helpful label (2.0%). Scores ranged from 2.60/10 (95% CI 1.68–4.04) for Mistral Medium 3 to 6.98/10 (95% CI 5.80–8.04) for Claude Sonnet 4.5, the only model with more helpful-only than harmful-only turns. Harmful behavior followed 74% of risky belief expressions and 73% of risky behavior expressions, versus 28% of turns without preceding risk expressions. Harmful responses often persisted: Claude recovered to helpful-only on 35% of transitions, versus 0–6% for four other models. Of 78 mixed responses, 77 combined a challenge to user beliefs or behaviors with harmful behavior. Performance varied substantially across personas.Discussion: Harmful responses frequently reinforced risky beliefs or behaviors, while support for independent reasoning was less common. Persistent harmful behavior, variable recovery, and mixed responses demonstrate why epistemic safety requires evaluation beyond factual accuracy or isolated outputs. Misinformation-related risks may arise through how models engage with users' beliefs over time. mPACT-MIS-v1.0 offers a behavioral framework for identifying potential reinforcement or correction, although it does not directly measure changes in users' beliefs or decisions.
Authors
- Alisa Breetz
- Shamal Lalvani (ORCID: https://orcid.org/0000-0002-2677-8019)
- Sarah Peregrine Lord (ORCID: https://orcid.org/0009-0006-2440-4108)
- Alison Cerezo (ORCID: https://orcid.org/0000-0003-4941-6616)
- Valerie Douglas (ORCID: https://orcid.org/0000-0003-0414-5373)
- Maggie L. Gorraiz
- Logan Lake
- Danielle A. Schlosser (ORCID: https://orcid.org/0000-0001-7652-0924)
- Paweł Świętojański (ORCID: https://orcid.org/0000-0001-5896-4505)
- Doğan Can (ORCID: https://orcid.org/0009-0005-5896-1351)
- Megan Greenlaw
- Caraline Bruzinski
- Kyle Etter
Publication Details
- Journal
- PsyArXiv (OSF Preprints)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.31234/osf.io/5c6vj_v2
- Primary Topic
- Misinformation and Its Impacts
- Type
- preprint