OCT-Trans: A Novel Transformer Backbone with Multimodal Feature Extraction in OCT-Based Retinal Disease Classification

Early detection of retinal diseases is vital for preventing irreversible vision loss. Unlike traditional machine learning (ML) approaches, this study introduces OCT-Trans, an innovative transformer-based backbone that significantly enhances the performance of ML classifiers in analyzing Optical Coherence Tomography (OCT) images. OCT-Trans demonstrates superior capability in distinguishing between normal retinas, Diabetic Retinopathy (DR), and Age-related Macular Degeneration (AMD). Our approach integrates multimodal features from OCT images acquired using two distinct imaging devices, thereby enhancing both generalizability and reproducibility. First, using an atlas-based method, we automatically segment OCT images into eleven retinal layers, extracting clinically relevant features that mimic ophthalmologists' diagnostics to get a final diagnosis. By examining each layer, our approach identifies critical indicators of DR and AMD such as the percentage of subretinal and sub-RPE fluid accumulation, abnormal tissue presence, and missing OCT layers. Additionally, we enrich our feature set by analyzing morphological and texture characteristics like estimating layer curvature, thickness, and reflectivity. All extracted features are integrated and fed into OCT-Trans, which utilizes a transformer architecture tailored for complex multimodal data. This innovative framework generates contextual embeddings that are subsequently classified using a multilayer perceptron. In a comprehensive evaluation involving 541 patients, OCT-Trans achieved remarkable results, with an accuracy of 96.68%, a Cohen's Kappa of 94.07%, and an F1 Score of 96.12%, significantly outperforming current state-of-the-art methods. Statistical analysis using Wilcoxon signed-rank tests confirmed significant enhancements <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(p &lt; 0.05)$</tex> over baseline architectures, including CaiT-T <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(p=0.001)$</tex>, ViT-B <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(p=0.001)$</tex>, and ConvNeXt-T <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(p=0.041)$</tex>.