AMCOS-Net: a novel attention-based multimodal fusion architecture for automated corneal opacity segmentation

Corneal opacity is one of the major causes of visual impairment across the world, and there is no objective way to diagnose it due to the subjectivity involved in the clinical assessment. To address this, the present study offers a multimodal deep learning framework which segments corneal opacity using anterior segment optical coherence tomography and anterior segment photography. This method utilizes the multimodal corneal opacity assessment dataset, which comprises 6664 multimodal images that provide both structural information of the cornea, and surface-level information. A number of segmentation baseline architectures, such as U-Net, U-Net++, Attention U-Net, V-Net, SegNet, and DeepLabV3+, were investigated with the use of early, mid, and late fusion strategies. It was found that mid-fusion level integration yielded the best results. Based on this, an architecture named AMCOS-Net was designed, which integrated dual-encoders, mid-level fusion with attention and atrous spatial pyramid pooling to capture context at multiple scales. AMCOS-Net has shown a small improvement in the quantitative results compared to all baseline models, with a mean segmentation Dice score of 0.584 and a mean intersection over union score of 0.412, and also showed a moderate improvement in the localization and segmentation of the corneal opacity boundaries. All of the above results could be considered as a preliminary study which has the potential for a multimodal fusion of two modalities that provide different pathological information about the cornea.