Generative AI Versus Primary Care Physicians from Plan Initiation to Long-Term Maintenance of Chronic Disease Self-Management: A Vignette-Based Validation Study

Background Generative artificial intelligence (GenAI) may provide personalised and dynamically adjustable support for chronic disease self-management. However, previous studies have largely focused on isolated questions and a limited range of evaluation dimensions. Objective To compare the quality of self-management advice generated by GenAI and primary care physicians (PCPs) across multiple stages of chronic disease management and evaluation dimensions. Methods We conducted a cross-sectional, source-blinded comparative evaluation using 15 standardised vignettes of diabetes and hypertension derived from authentic primary care follow-up records. Each vignette comprised three sequential management stages, with medical accuracy, readability, and behaviour-change potential assessed across the three simulated stages; individualisation and multidisciplinary integration at plan initiation; contextual adaptability during acute adjustment; and empathy during long-term maintenance. The GenAI system and 22 PCPs independently generated advice which was independently rated by three clinicians,one each from general practice, endocrinology, and cardiology. across seven dimensions. Linear mixed-effects models were used for the primary comparisons, and case-level paired and physician-distribution analyses were conducted as robustness analyses. Results The GenAI system achieved estimated marginal means above 4.0 in all dimensions except readability. It scored significantly higher than PCPs in six of the seven dimensions, with adjusted mean differences ranging from 1.16 to 1.56 points (P<.001). Readability was the only exception, with no significant difference between GenAI and PCPs (−0.22, 95% CI −0.54 to 0.11; P=.19). The GenAI mean exceeded the individual means of all recruited PCPs in the six dimensions but ranked at approximately the 23rd percentile for readability. Conclusions Our findings indicated the GenAI system may generate more highly rated self-management advice than PCPs across most evaluated dimensions but offered no advantage in readability. These findings provide pre-implementation evidence for GenAI as a potential source of structured and adaptive chronic disease self-management support. Patient-level pilot studies and subsequent randomised trials are needed to determine its acceptability and effects on behaviour and clinical outcomes.

Authors

Institutions

Publication Details

Journal
medRxiv
Published
2026-11-12
DOI
https://doi.org/10.64898/2026.11.10.26367736
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Generative AI Versus Primary Care Physicians from Plan Initiation to Long-Term Maintenance of Chronic Disease Self-Management: A Vignette-Based Validation Study

马晓晨, Ziyin Zhao, Ying Wang, Qiting Wang et al.
medRxiv
Artificial Intelligence in Healthcare and Education
preprint

Generative AI Versus Primary Care Physicians from Plan Initiation to Long-Term Maintenance of Chronic Disease Self-Management: A Vignette-Based Validation Study

马晓晨, Ziyin Zhao, Ying Wang, Qiting Wang, Weibin Cheng, Huiming Yang
preprint en

Abstract

Background Generative artificial intelligence (GenAI) may provide personalised and dynamically adjustable support for chronic disease self-management. However, previous studies have largely focused on isolated questions and a limited range of evaluation dimensions. Objective To compare the quality of self-management advice generated by GenAI and primary care physicians (PCPs) across multiple stages of chronic disease management and evaluation dimensions. Methods We conducted a cross-sectional, source-blinded comparative evaluation using 15 standardised vignettes of diabetes and hypertension derived from authentic primary care follow-up records. Each vignette comprised three sequential management stages, with medical accuracy, readability, and behaviour-change potential assessed across the three simulated stages; individualisation and multidisciplinary integration at plan initiation; contextual adaptability during acute adjustment; and empathy during long-term maintenance. The GenAI system and 22 PCPs independently generated advice which was independently rated by three clinicians,one each from general practice, endocrinology, and cardiology. across seven dimensions. Linear mixed-effects models were used for the primary comparisons, and case-level paired and physician-distribution analyses were conducted as robustness analyses. Results The GenAI system achieved estimated marginal means above 4.0 in all dimensions except readability. It scored significantly higher than PCPs in six of the seven dimensions, with adjusted mean differences ranging from 1.16 to 1.56 points (P<.001). Readability was the only exception, with no significant difference between GenAI and PCPs (−0.22, 95% CI −0.54 to 0.11; P=.19). The GenAI mean exceeded the individual means of all recruited PCPs in the six dimensions but ranked at approximately the 23rd percentile for readability. Conclusions Our findings indicated the GenAI system may generate more highly rated self-management advice than PCPs across most evaluated dimensions but offered no advantage in readability. These findings provide pre-implementation evidence for GenAI as a potential source of structured and adaptive chronic disease self-management support. Patient-level pilot studies and subsequent randomised trials are needed to determine its acceptability and effects on behaviour and clinical outcomes.

medRxiv
Sun Yat-sen University (CN), Guangdong Pharmaceutical University (CN), Peking University (CN), Guangdong Provincial People's Hospital (CN)
National Natural Science Foundation of China
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon