mPACT-MIS-v1.0: A Clinician–Informed Framework for Evaluating Epistemic Drift in Large Language Models

Introduction: Large language models (LLMs) increasingly shape how users evaluate information, form beliefs, and make decisions. Epistemic quality refers to the extent to which responses support accurate understanding, calibrated confidence, evidence-grounded beliefs, and independent reasoning. Risks extend beyond misinformation to reinforcing unsupported beliefs, expressing unwarranted certainty, and failing to challenge misleading assumptions. We developed mPACT-MIS-v1.0 to evaluate model behaviors that may support or undermine epistemic quality.Methods: We evaluated 60 expert-authored, multi-turn conversations (642 model turns; mean 10.70 [SD 2.70], range 5–17) across ten personas and six LLMs: Claude Sonnet 4.5, Gemini (2.5 Flash/3 Flash Preview), GPT-5.2, Grok 4.1, LLaMA 3.3 70B, and Mistral Medium 3. Trained annotators coded model turns using four helpful and four harmful behavioral labels informed by Self-Determination Theory and user turns for risky belief or behavior expressions. Model responses were classified as helpful-only, harmful-only, mixed, or uncoded. Conversation-averaged scores (0–10; higher is better) were calculated with 95% confidence intervals from conversation-clustered bootstrapping. Additional analyses examined responses to risk, transitions, correction latency, and persona variation.Results: Of 642 model turns, 261 (40.7%) were harmful-only, 78 (12.1%) mixed, 139 (21.7%) helpful-only, and 164 (25.5%) uncoded; 52.8% contained harmful behavior. Promoting risky beliefs or behaviors was the most frequent harmful label, while assessing risk was the least frequent helpful label (2.0%). Scores ranged from 2.60/10 (95% CI 1.68–4.04) for Mistral Medium 3 to 6.98/10 (95% CI 5.80–8.04) for Claude Sonnet 4.5, the only model with more helpful-only than harmful-only turns. Harmful behavior followed 74% of risky belief expressions and 73% of risky behavior expressions, versus 28% of turns without preceding risk expressions. Harmful responses often persisted: Claude recovered to helpful-only on 35% of transitions, versus 0–6% for four other models. Of 78 mixed responses, 77 combined a challenge to user beliefs or behaviors with harmful behavior. Performance varied substantially across personas.Discussion: Harmful responses frequently reinforced risky beliefs or behaviors, while support for independent reasoning was less common. Persistent harmful behavior, variable recovery, and mixed responses demonstrate why epistemic safety requires evaluation beyond factual accuracy or isolated outputs. Misinformation-related risks may arise through how models engage with users' beliefs over time. mPACT-MIS-v1.0 offers a behavioral framework for identifying potential reinforcement or correction, although it does not directly measure changes in users' beliefs or decisions.

Authors

Publication Details

Journal
PsyArXiv (OSF Preprints)
Published
2026-11-19
DOI
https://doi.org/10.31234/osf.io/5c6vj
Primary Topic
Misinformation and Its Impacts
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

mPACT-MIS-v1.0: A Clinician–Informed Framework for Evaluating Epistemic Drift in Large Language Models

Alisa Breetz, Shamal Lalvani, Sarah Peregrine Lord, Alison Cerezo et al.
PsyArXiv (OSF Preprints)
Misinformation and Its Impacts
preprint

mPACT-MIS-v1.0: A Clinician–Informed Framework for Evaluating Epistemic Drift in Large Language Models

Alisa Breetz, Shamal Lalvani, Sarah Peregrine Lord, Alison Cerezo, Valerie Douglas, Maggie L. Gorraiz, Logan Lake, Danielle A. Schlosser, Paweł Świętojański, Doğan Can, Megan Greenlaw, Caraline Bruzinski, Kyle Etter
preprint en

Abstract

Introduction: Large language models (LLMs) increasingly shape how users evaluate information, form beliefs, and make decisions. Epistemic quality refers to the extent to which responses support accurate understanding, calibrated confidence, evidence-grounded beliefs, and independent reasoning. Risks extend beyond misinformation to reinforcing unsupported beliefs, expressing unwarranted certainty, and failing to challenge misleading assumptions. We developed mPACT-MIS-v1.0 to evaluate model behaviors that may support or undermine epistemic quality.Methods: We evaluated 60 expert-authored, multi-turn conversations (642 model turns; mean 10.70 [SD 2.70], range 5–17) across ten personas and six LLMs: Claude Sonnet 4.5, Gemini (2.5 Flash/3 Flash Preview), GPT-5.2, Grok 4.1, LLaMA 3.3 70B, and Mistral Medium 3. Trained annotators coded model turns using four helpful and four harmful behavioral labels informed by Self-Determination Theory and user turns for risky belief or behavior expressions. Model responses were classified as helpful-only, harmful-only, mixed, or uncoded. Conversation-averaged scores (0–10; higher is better) were calculated with 95% confidence intervals from conversation-clustered bootstrapping. Additional analyses examined responses to risk, transitions, correction latency, and persona variation.Results: Of 642 model turns, 261 (40.7%) were harmful-only, 78 (12.1%) mixed, 139 (21.7%) helpful-only, and 164 (25.5%) uncoded; 52.8% contained harmful behavior. Promoting risky beliefs or behaviors was the most frequent harmful label, while assessing risk was the least frequent helpful label (2.0%). Scores ranged from 2.60/10 (95% CI 1.68–4.04) for Mistral Medium 3 to 6.98/10 (95% CI 5.80–8.04) for Claude Sonnet 4.5, the only model with more helpful-only than harmful-only turns. Harmful behavior followed 74% of risky belief expressions and 73% of risky behavior expressions, versus 28% of turns without preceding risk expressions. Harmful responses often persisted: Claude recovered to helpful-only on 35% of transitions, versus 0–6% for four other models. Of 78 mixed responses, 77 combined a challenge to user beliefs or behaviors with harmful behavior. Performance varied substantially across personas.Discussion: Harmful responses frequently reinforced risky beliefs or behaviors, while support for independent reasoning was less common. Persistent harmful behavior, variable recovery, and mixed responses demonstrate why epistemic safety requires evaluation beyond factual accuracy or isolated outputs. Misinformation-related risks may arise through how models engage with users' beliefs over time. mPACT-MIS-v1.0 offers a behavioral framework for identifying potential reinforcement or correction, although it does not directly measure changes in users' beliefs or decisions.

PsyArXiv (OSF Preprints)
Misinformation and Its Impacts
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon