Large language models for diabetes training: a prospective study

Diabetes poses a considerable global health challenge, with varying levels of diabetes knowledge among healthcare professionals, highlighting the importance of diabetes training. Large Language Models (LLMs) provide new insights into diabetes training, but their performance in diabetes-related queries remains uncertain, especially outside the English language like Chinese. We first evaluated the performance of ten LLMs: ChatGPT-3.5, ChatGPT-4.0, Google Bard, LlaMA-7B, LlaMA2-7B, Baidu ERNIE Bot, Ali Tongyi Qianwen, MedGPT, HuatuoGPT, and Chinese LlaMA2-7B on diabetes-related queries, based on the Chinese National Certificate Examination for Primary Diabetes Care in China (NCE-CPDC) and the English Specialty Certificate Examination in Endocrinology and Diabetes of Membership of the Royal College of Physicians of the United Kingdom. Second, we assessed the training of primary care physicians (PCPs) without and with the assistance of ChatGPT-4.0 in the NCE-CPDC examination to ascertain the reliability of LLMs as medical assistants. We found that ChatGPT-4.0 outperformed other LLMs in the English examination, achieving a passing accuracy of 62.50%, which was significantly higher than that of Google Bard, LlaMA-7B, and LlaMA2-7B. For the NCE-CPFC examination, ChatGPT-4.0, Ali Tongyi Qianwen, Baidu ERNIE Bot, Google Bard, MedGPT, and ChatGPT-3.5 successfully passed, whereas LlaMA2-7B, HuatuoGPT, Chinese LLaMA2-7B, and LlaMA-7B failed. ChatGPT-4.0 (84.82%) surpassed all PCPs and assisted most PCPs in the NCE-CPDC examination (improving by 1 %-6.13%). In summary, LLMs demonstrated outstanding competence for diabetes-related questions in both the Chinese and English language, and hold great potential to assist future diabetes training for physicians globally.

Authors

Institutions

Publication Details

Journal
Science Bulletin
Published
2025-01-27
DOI
https://doi.org/10.1016/j.scib.2025.01.034
Citations
16
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
10.47

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Large language models for diabetes training: a prospective study

Zhouyu Guan, Elif Ilhan Ekinci, Weiping Jia, Di Cheng et al.
16 citations
Science Bulletin
Artificial Intelligence in Healthcare and Education
10.47
article

Large language models for diabetes training: a prospective study

Zhouyu Guan, Elif Ilhan Ekinci, Weiping Jia, Di Cheng, Yingfeng Zheng, Charumathi Sabanayagam, Yuqian Bao, Ya Xing Wang, Huating Li, Carol Y. Cheung, Ruhan Liu, Justin Basile Echouffo-Tcheugui, Ching‐Yu Cheng, Hongwei Ji, Liang Wu, Zehua Jiang, Leonor Corsino, Cynthia Ciwei Lim, Nestoras Mathioudakis, Yong Mong Bee, Hong Chang Tan, Yuexing Liu, Tien Yin Wong, Gavin Siew Wei Tan, Yih Chung Tham, David Charles Klonoff, Weibing Wu, Lee‐Ling Lim, Rafael Simó, Yong Wang, Jiajia Li, Haoxuan Li, Jihong Wang, Zhen Li, Chun Cai, Bin Sheng, Lijuan Mao, Tingting Hu
article en
16 citations

Abstract

Diabetes poses a considerable global health challenge, with varying levels of diabetes knowledge among healthcare professionals, highlighting the importance of diabetes training. Large Language Models (LLMs) provide new insights into diabetes training, but their performance in diabetes-related queries remains uncertain, especially outside the English language like Chinese. We first evaluated the performance of ten LLMs: ChatGPT-3.5, ChatGPT-4.0, Google Bard, LlaMA-7B, LlaMA2-7B, Baidu ERNIE Bot, Ali Tongyi Qianwen, MedGPT, HuatuoGPT, and Chinese LlaMA2-7B on diabetes-related queries, based on the Chinese National Certificate Examination for Primary Diabetes Care in China (NCE-CPDC) and the English Specialty Certificate Examination in Endocrinology and Diabetes of Membership of the Royal College of Physicians of the United Kingdom. Second, we assessed the training of primary care physicians (PCPs) without and with the assistance of ChatGPT-4.0 in the NCE-CPDC examination to ascertain the reliability of LLMs as medical assistants. We found that ChatGPT-4.0 outperformed other LLMs in the English examination, achieving a passing accuracy of 62.50%, which was significantly higher than that of Google Bard, LlaMA-7B, and LlaMA2-7B. For the NCE-CPFC examination, ChatGPT-4.0, Ali Tongyi Qianwen, Baidu ERNIE Bot, Google Bard, MedGPT, and ChatGPT-3.5 successfully passed, whereas LlaMA2-7B, HuatuoGPT, Chinese LLaMA2-7B, and LlaMA-7B failed. ChatGPT-4.0 (84.82%) surpassed all PCPs and assisted most PCPs in the NCE-CPDC examination (improving by 1 %-6.13%). In summary, LLMs demonstrated outstanding competence for diabetes-related questions in both the Chinese and English language, and hold great potential to assist future diabetes training for physicians globally.

Science BulletinVol. 70(6)
Qingdao University (CN), Sun Yat-sen University (CN), Duke University (US), Chinese University of Hong Kong (HK), Shanghai University of Sport (CN), Singapore General Hospital (SG), Singapore National Eye Center (SG), Affiliated Hospital of Qingdao University (CN), Singapore Eye Research Institute (SG), Mills Peninsula Health Services (US)
National Natural Science Foundation of China
Quality Education
Openalex Percentile: Top 2%
Artificial Intelligence in Healthcare and Education
10.47
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.