A deep learning based automatic report generator for retinal optical coherence tomography images
Reading and summarizing insights from Optical Coherence Tomography (OCT) images is a routine yet time-consuming task that requires expensive time from experienced ophthalmologists. This paper introduces the Multi-label OCT Report Generation (MORG) model, a deep learning approach to assist in the interpretation of OCT images. MORG employs dual image encoders to extract features from OCT image pairs, fusing them through a multi-scale module with an attention mechanism, followed by a sentence decoder to produce reports. Trained and tested on 57,308 retinal OCT image pairs, MORG achieved high classification accuracy for 16 pathologies with 37 descriptive types. It also excelled in a blind grading test against general large language models and other state-of-the-art image captioning models, scoring 4.55 compared to ophthalmologists' 4.63 out of a maximum of 5. Furthermore, MORG has the potential to reduce the report drafting time for ophthalmologists by 58.9%, significantly alleviating their workload.
Authors
- Fei Shi (ORCID: https://orcid.org/0000-0003-1545-3494)
- Cangxin Li (ORCID: https://orcid.org/0009-0005-4022-0563)
- Huazhu Fu (ORCID: https://orcid.org/0000-0002-9702-5524)
- Baoqing Nie (ORCID: https://orcid.org/0000-0003-4631-0728)
- Tian Lin (ORCID: https://orcid.org/0000-0002-6415-5381)
- Dehui Xiang (ORCID: https://orcid.org/0000-0001-7873-9778)
- Yuanyuan Peng (ORCID: https://orcid.org/0000-0003-2852-1951)
- Anlin Zhang (ORCID: https://orcid.org/0009-0004-2076-753X)
- Shirong Chen (ORCID: https://orcid.org/0000-0002-2159-8844)
- Danqi Fang
- Weifang Zhu (ORCID: https://orcid.org/0000-0001-9540-4101)
- Ting Wang (ORCID: https://orcid.org/0000-0002-7223-8849)
- Chunguang Guo (ORCID: https://orcid.org/0000-0002-3361-9411)
- Chi Pui Pang (ORCID: https://orcid.org/0000-0002-7764-6382)
- Zhongyue Chen (ORCID: https://orcid.org/0000-0001-5097-0812)
- Mingzhi Zhang (ORCID: https://orcid.org/0000-0001-9032-7274)
- Meng Wang (ORCID: https://orcid.org/0000-0001-7882-1747)
- Jiangtao Wang (ORCID: https://orcid.org/0000-0002-8704-502X)
- Yi Zhou (ORCID: https://orcid.org/0000-0003-3021-3229)
- Qian Cheng (ORCID: https://orcid.org/0000-0003-3621-5743)
- Haoyu Chen (ORCID: https://orcid.org/0000-0001-7618-9733)
- Xinjian Chen (ORCID: https://orcid.org/0000-0001-5770-5640)
- Aidi Lin
Institutions
- Agency for Science, Technology and Research (SG)
- University of Electronic Science and Technology of China (CN)
- Chinese University of Hong Kong (HK)
- Institute of High Performance Computing (SG)
- Shantou University (CN)
- Soochow University (CN)
Publication Details
- Journal
- npj Digital Medicine
- Published
- 2025-10-20
- DOI
- https://doi.org/10.1038/s41746-025-01988-2
- Primary Topic
- Retinal Imaging and Analysis
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- Agency for Science, Technology and Research
- National Natural Science Foundation of China
- Government of Jiangsu Province
- Natural Science Foundation of Jiangsu Province
- Department of Education of Guangdong Province