A multimodal visual–language foundation model for computational ophthalmology

Early detection of eye diseases is vital for preventing vision loss. Existing ophthalmic artificial intelligence models focus on single modalities, overlooking multi-view information and struggling with rare diseases due to long-tail distributions. We propose EyeCLIP, a multimodal visual-language foundation model trained on 2.77 million ophthalmology images from 11 modalities with partial clinical text. Our novel pretraining strategy combines self-supervised reconstruction, multimodal image contrastive learning, and image-text contrastive learning to capture shared representations across modalities. EyeCLIP demonstrates robust performance across 14 benchmark datasets, excelling in disease classification, visual question answering, and cross-modal retrieval. It also exhibits strong few-shot and zero-shot capabilities, enabling accurate predictions in real-world, long-tail scenarios. EyeCLIP offers significant potential for detecting both ocular and systemic diseases, and bridging gaps in real-world clinical applications.

Authors

Institutions

Publication Details

Journal
npj Digital Medicine
Published
2025-06-21
DOI
https://doi.org/10.1038/s41746-025-01772-2
Citations
29
Primary Topic
Retinal Imaging and Analysis
Type
article
Field-Weighted Citation Impact
39.44
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

A multimodal visual–language foundation model for computational ophthalmology

Mingguang He, Kai Jin, Siyu Huang, Danli Shi et al.
29 citations
npj Digital Medicine
Retinal Imaging and Analysis
39.44
article

A multimodal visual–language foundation model for computational ophthalmology

Mingguang He, Kai Jin, Siyu Huang, Danli Shi, Pusheng Xu, Zongyuan Ge, J. Yang, Shunming Liu, Q Zhang, Mayinuer Yusufu, Xun Xu, Xiaolan Chen, Shan C. Lin, Wei Jin, Weiyi Zhang
article en
29 citations

Abstract

Early detection of eye diseases is vital for preventing vision loss. Existing ophthalmic artificial intelligence models focus on single modalities, overlooking multi-view information and struggling with rare diseases due to long-tail distributions. We propose EyeCLIP, a multimodal visual-language foundation model trained on 2.77 million ophthalmology images from 11 modalities with partial clinical text. Our novel pretraining strategy combines self-supervised reconstruction, multimodal image contrastive learning, and image-text contrastive learning to capture shared representations across modalities. EyeCLIP demonstrates robust performance across 14 benchmark datasets, excelling in disease classification, visual question answering, and cross-modal retrieval. It also exhibits strong few-shot and zero-shot capabilities, enabling accurate predictions in real-world, long-tail scenarios. EyeCLIP offers significant potential for detecting both ocular and systemic diseases, and bridging gaps in real-world clinical applications.

npj Digital MedicineVol. 8(1)
Beijing Tongren Hospital (CN), Hong Kong Polytechnic University (HK), Capital Medical University (CN), Shanghai Eye Disease Prevention & Treatment Center (CN), Centre for Eye Research Australia (AU), Guangdong Academy of Medical Sciences (CN), Guangdong Provincial People's Hospital (CN), Second Affiliated Hospital of Zhejiang University (CN), The Eighth Hospital of Xi'an (CN), École Polytechnique Fédérale de Lausanne (CH), Monash University (AU), Clemson University (US)
Quality Education
Openalex Percentile: Top 0%
Retinal Imaging and Analysis
39.44
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.