Quality, reliability, and readability of large language model responses to questions about immune checkpoint inhibitor-associated myocarditis: A cross-sectional study

Immune checkpoint inhibitor-associated myocarditis (ICI-M) is a rare but often fatal immune-related adverse event. Large language models (LLMs) have provided new opportunities for patient education, yet whether they can effectively support patient education in ICI-M remains unknown. In this study, we submitted 20 standardized ICI-M-related patient questions to five LLMs (ChatGPT-5.5, Gemini-3.5 Flash, DeepSeek-V4, Qwen 3.7, and Doubao), generating 100 responses. Model performance was evaluated using the Global Quality Score (GQS), modified DISCERN (mDISCERN), Quality Analysis of Medical Artificial Intelligence (QAMAI), Ensuring Quality Information for Patients (EQIP), Patient Education Materials Assessment Tool understandability and actionability scores (PEMAT-U and PEMAT-A), and six readability indices. Significant inter-model differences were observed across most quality and readability metrics, and no single model outperformed the others across all dimensions. DeepSeek-V4 achieved the highest mDISCERN score [5.00 (IQR, 4.00–5.00)], whereas ChatGPT-5.5 achieved the highest PEMAT-A score [40.00 (IQR, 35.00–45.00)]. Readability was suboptimal, with a mean FKGL of 16.60 ± 2.64. Correlation analyses demonstrated information quality, usability, and readability represented correlated but distinct evaluation dimensions. While LLMs may serve as adjunctive tools for ICI-M patient education, substantial optimization of content accessibility and actionability is required prior to clinical implementation.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-11077152/v1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Quality, reliability, and readability of large language model responses to questions about immune checkpoint inhibitor-associated myocarditis: A cross-sectional study

Danfeng Dong, Yan Zhou, Lei Wu, Juan Zhang et al.
4 links
Research Square
Artificial Intelligence in Healthcare and Education
preprint

Quality, reliability, and readability of large language model responses to questions about immune checkpoint inhibitor-associated myocarditis: A cross-sectional study

Danfeng Dong, Yan Zhou, Lei Wu, Juan Zhang, Mi Zhang, Jin Yang, Tongfei Wang
preprint en

Abstract

Immune checkpoint inhibitor-associated myocarditis (ICI-M) is a rare but often fatal immune-related adverse event. Large language models (LLMs) have provided new opportunities for patient education, yet whether they can effectively support patient education in ICI-M remains unknown. In this study, we submitted 20 standardized ICI-M-related patient questions to five LLMs (ChatGPT-5.5, Gemini-3.5 Flash, DeepSeek-V4, Qwen 3.7, and Doubao), generating 100 responses. Model performance was evaluated using the Global Quality Score (GQS), modified DISCERN (mDISCERN), Quality Analysis of Medical Artificial Intelligence (QAMAI), Ensuring Quality Information for Patients (EQIP), Patient Education Materials Assessment Tool understandability and actionability scores (PEMAT-U and PEMAT-A), and six readability indices. Significant inter-model differences were observed across most quality and readability metrics, and no single model outperformed the others across all dimensions. DeepSeek-V4 achieved the highest mDISCERN score [5.00 (IQR, 4.00–5.00)], whereas ChatGPT-5.5 achieved the highest PEMAT-A score [40.00 (IQR, 35.00–45.00)]. Readability was suboptimal, with a mean FKGL of 16.60 ± 2.64. Correlation analyses demonstrated information quality, usability, and readability represented correlated but distinct evaluation dimensions. While LLMs may serve as adjunctive tools for ICI-M patient education, substantial optimization of content accessibility and actionability is required prior to clinical implementation.

Research Square
Northwest University (CN), Northwest Hospital and Medical Center (US), First Affiliated Hospital of Xi'an Jiaotong University (CN)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon