Multiple attention based deep multimodal fusion network for glaucoma and neurodegenerative disease diagnosis

Glaucoma and other neurodegenerative diseases involve progressive damage to the optic disc (OD) and can be identified using visual field (VF) tests, colour fundus photographs (CFP) and optical coherence tomography (OCT). However, most studies overlook neurodegenerative cohorts and primarily rely on unimodal approaches, thus failing to leverage advanced deep multimodal fusion techniques to capture modality-specific features and cross-modal interactions. To address the issue, this study aims to design a multi-attention-based deep multimodal network to fuse functional VF with structural CFP and OCT. Using data from 354 eyes with neuropathies (glaucoma, ischaemic optic neuropathy, dementia and Parkinson’s) and 228 normal eyes, we developed novel deep learning techniques for disc segmentation, fovea localisation, disc-fovea angle calculation and alignment, and multi-attention-based deep intermediate multimodal fusion incorporating different combinations of spatial, self- and cross-modal attention mechanisms. The results indicate that our proposed multi-attention-multimodal fusion network (MAM-Fusion-Net) achieved an area under the sensitivity–specificity curve (AUC) of 0.998 (95% CI 0.996–1.000) for diagnosing neuropathy versus normal participants and 0.981 (95% CI 0.966–0.996) for glaucoma versus non-glaucoma, surpassing state-of-the-art uni- and multi-modal models. This multi-attention-based deep fusion strategy applied to multimodal images shows potential in glaucoma and neurodegenerative disease diagnosis to assist eye care practitioners.