Enhancing AI reliability: A foundation model with uncertainty estimation for optical coherence tomography-based retinal disease diagnosis
Inability to express the confidence level and detect unseen disease classes limits the clinical implementation of artificial intelligence in the real world. We develop a foundation model with uncertainty estimation (FMUE) to detect 16 retinal conditions on optical coherence tomography (OCT). In the internal test set, FMUE achieves a higher F1 score of 95.74% than other state-of-the-art algorithms (92.03%-93.66%) and improves to 97.44% with threshold strategy. The model achieves similar excellent performance on two external test sets from the same and different OCT machines. In human-model comparison, FMUE achieves a higher F1 score of 96.30% than retinal experts (86.95%, p = 0.004), senior doctors (82.71%, p < 0.001), junior doctors (66.55%, p < 0.001), and generative pretrained transformer 4 with vision (GPT-4V) (32.39%, p < 0.001). Besides, FMUE predicts high uncertainty scores for >85% images of non-target-category diseases or with low quality to prompt manual checks and prevent misdiagnosis. Our FMUE provides a trustworthy method for automatic retinal anomaly detection in a clinical open-set environment.
Authors
- Shanshan Yu (ORCID: https://orcid.org/0000-0001-6182-8328)
- Dawei Sun (ORCID: https://orcid.org/0000-0002-9717-0190)
- Yuanyuan Peng (ORCID: https://orcid.org/0000-0003-2852-1951)
- Tingkun Shi (ORCID: https://orcid.org/0000-0003-2382-817X)
- Ke Zou (ORCID: https://orcid.org/0000-0002-1181-1779)
- Xinjian Chen (ORCID: https://orcid.org/0000-0002-0871-293X)
- Tao Li (ORCID: https://orcid.org/0000-0003-1697-8022)
- Jian Wu (ORCID: https://orcid.org/0000-0002-3394-1507)
- Ling Gao (ORCID: https://orcid.org/0000-0002-6391-6724)
- Lixia Feng (ORCID: https://orcid.org/0000-0002-1910-8068)
- Tian Lin
- Dan Liang (ORCID: https://orcid.org/0000-0002-8722-9144)
- Linna Liu
- Meng Wang (ORCID: https://orcid.org/0000-0001-9590-4202)
- Aidi Lin
- Jing Luo (ORCID: https://orcid.org/0009-0008-3540-4191)
- Zhen Liang (ORCID: https://orcid.org/0000-0002-5242-4644)
Institutions
- Central South University (CN)
- Harbin Medical University (CN)
- Sun Yat-sen University (CN)
- National University of Singapore (SG)
- Chinese University of Hong Kong (HK)
- Anhui Medical University (CN)
- Sichuan University (CN)
- Singapore National Eye Center (SG)
- Institute of High Performance Computing (SG)
- Shantou University (CN)
- Soochow University (CN)
- Chuzhou University (CN)
- Second Affiliated Hospital of Harbin Medical University (CN)
- First Affiliated Hospital of Anhui Medical University (CN)
- Second Xiangya Hospital of Central South University (CN)
- He Eye Hospital (CN)
Publication Details
- Journal
- Cell Reports Medicine
- Published
- 2024-12-19
- DOI
- https://doi.org/10.1016/j.xcrm.2024.101876
- Citations
- 11
- Primary Topic
- Retinal Imaging and Analysis
- Type
- article
- Field-Weighted Citation Impact
- 4.45
Funders
- Agency for Science, Technology and Research
- National Natural Science Foundation of China
- Department of Education of Guangdong Province
- University Natural Science Research Project of Anhui Province