Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses

Chatbot-based multimodal AI holds promise for collecting medical histories and diagnosing ophthalmic diseases using textual and imaging data. This study developed and evaluated the ChatGPT-powered Intelligent Ophthalmic Multimodal Interactive Diagnostic System (IOMIDS) to enable patient self-diagnosis and self-triage. IOMIDS included a text model and three multimodal models (text + slit-lamp, text + smartphone, text + slit-lamp + smartphone). The performance was evaluated through a two-stage cross-sectional study across three medical centers involving 10 subspecialties and 50 diseases. Using 15640 data entries, IOMIDS actively collected and analyzed medical history alongside slit-lamp and/or smartphone images. The text + smartphone model showed the highest diagnostic accuracy (internal: 79.6%, external: 81.1%), while other multimodal models underperformed or matched the text model (internal: 69.6%, external: 72.5%). Moreover, triage accuracy was consistent across models. Multimodal approaches enhanced response quality and reduced misinformation. This proof-of-concept study highlights the potential of chatbot-based multimodal AI for self-diagnosis and self-triage. (The clinical trial was registered on June 26, 2023, on ClinicalTrials.gov under the registration number NCT05930444.).

Authors

Institutions

Publication Details

Journal
npj Digital Medicine
Published
2025-01-27
DOI
https://doi.org/10.1038/s41746-025-01461-0
Citations
27
Primary Topic
Retinal Imaging and Analysis
Type
article
Field-Weighted Citation Impact
36.72

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses

Qian Cheng, Yue Zhao, Jingjing Liao, Baoqing Nie et al.
27 citations
npj Digital Medicine
Retinal Imaging and Analysis
36.72
article

Multimodal machine learning enables AI chatbot to diagnose ophthalmic diseases and provide high-quality medical responses

Qian Cheng, Yue Zhao, Jingjing Liao, Baoqing Nie, Xintong Lin, Hongguang Cui, Bing Qin, Xiawei Wang, Yunqiu Zhang, Dehui Xiang, Zhiyu Peng, Xiangjia Zhu, Jin-Wei Cheng, Fei Shi, Weifang Zhu, Wenjun Shu, Jiaxu Hong, Pengfei Jiang, Weiyi Xia, Xinjian Chen, Jie Guo, Jie Lu, Lu Gan, Haifeng Chen, Jiangtao Wang, Ruiqi Ma, Chen Zhao, Jiang Zhu, Mingxu Yan, Qin Jiang, Jinghan Wang, Jing Yao, Kang Xue, Lejin Tian, Xiaofeng Li, Rui Zhang
article en
27 citations

Abstract

Chatbot-based multimodal AI holds promise for collecting medical histories and diagnosing ophthalmic diseases using textual and imaging data. This study developed and evaluated the ChatGPT-powered Intelligent Ophthalmic Multimodal Interactive Diagnostic System (IOMIDS) to enable patient self-diagnosis and self-triage. IOMIDS included a text model and three multimodal models (text + slit-lamp, text + smartphone, text + slit-lamp + smartphone). The performance was evaluated through a two-stage cross-sectional study across three medical centers involving 10 subspecialties and 50 diseases. Using 15640 data entries, IOMIDS actively collected and analyzed medical history alongside slit-lamp and/or smartphone images. The text + smartphone model showed the highest diagnostic accuracy (internal: 79.6%, external: 81.1%), while other multimodal models underperformed or matched the text model (internal: 69.6%, external: 72.5%). Moreover, triage accuracy was consistent across models. Multimodal approaches enhanced response quality and reduced misinformation. This proof-of-concept study highlights the potential of chatbot-based multimodal AI for self-diagnosis and self-triage. (The clinical trial was registered on June 26, 2023, on ClinicalTrials.gov under the registration number NCT05930444.).

npj Digital MedicineVol. 8(1)
Shanghai Medical College of Fudan University (CN), Shanghai University of Engineering Science (CN), Shanghai Jiao Tong University (CN), Chinese Academy of Medical Sciences & Peking Union Medical College (CN), Fudan University (CN), Soochow University (CN), Eye & ENT Hospital of Fudan University (CN), Second Affiliated Hospital of Nanjing Medical University (CN), Shandong Eye Hospital (CN), First Affiliated Hospital Zhejiang University (CN), State Key Laboratory of Genetic Engineering, Nanjing Medical University (CN)
National Natural Science Foundation of China, Government of Jiangsu Province, Fudan University, Natural Science Foundation of Jiangsu Province, Zhejiang University, Shanghai Jiao Tong University, Chinese Academy of Medical Sciences, Nanjing Medical University, Soochow University, Shanghai Hospital Development Center, State Key Laboratory of Genetic Engineering, State Key Laboratory of Radiation Medicine and Protection
Openalex Percentile: Top 0%
Retinal Imaging and Analysis
36.72
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.