Diagnosis of retinal diseases driven by Mamba multi-modal CLIP auxiliary fusion network

The high prevalence of retinal diseases has made early diagnosis a critical focus in ophthalmology research. Owing to the diversity of imaging modalities, fundus data often exhibit significant modality-specific differences, with each modality offering unique strengths in identifying particular retinal conditions. To address this, we propose M2AF-Net, a multimodal CLIP auxiliary fusion network powered by the Mamba architecture, for improved retinal disease diagnosis. Specifically, we design modality-specific Mamba-based feature extractors that separately capture the global context from color fundus photography (CFP) and hierarchical structural details from optical coherence tomography (OCT). To enrich feature representation, we introduce an auxiliary diagnostic pathway using a pre-trained CLIP encoder, which extracts cross-modal semantic correlations from preprocessed CFP and OCT images, providing complementary diagnostic cues. Furthermore, we propose a cross-modal Mamba fusion (CMMF) module that integrates a loose pairwise training strategy with an adaptive dynamic weighting mechanism to optimize modality contributions. This module leverages Mamba-based alignment to enhance fusion efficiency and overall diagnostic performance. Extensive experiments on challenging benchmark datasets demonstrate the effectiveness of our approach in accurately diagnosing a wide range of retinal diseases. The code for this project is available at https://github.com/Skyler-Morgan/M2AFNet . • Propose a Mamba-based multi-modal diagnostic framework for retinal diseases. • Design modality-specific extractors for CFP and OCT feature representation. • Integrate CLIP for semantic guidance to enhance lesion feature extraction. • Introduce dynamic Mamba fusion for deep cross-modal alignment.