Multimodal patch-based deep learning for early Alzheimer’s disease classification using MRI and PET

Alzheimer’s disease (AD) is a neurodegenerative condition in which detection of pathological changes in the early stages, especially during the mild cognitive impairment (MCI) phase, is key for early diagnosis and treatment of the disease. While MRI gives a structural view of the brain, FDG-PET is complementary in providing a view of functional and metabolic changes in the brain due to neurodegeneration. In this study, we propose a multimodal patch-based deep learning framework for classification of AD, MCI, and cognitively normal (NC) patients based on MRI and FDG-PET data pairs. Data consisted of 168 patients from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database, which include 58 AD, 54 MCI, and 56 NC patients. T1-weighted MRI and FDG-PET volumes were preprocessed and registered to a common spatial space, followed by extraction of non-overlapping 3-D patches. We then extracted structural and functional features using two 3D convolutional neural networks (3D-CNNs), and finally multimodal fusion of features was performed. In order to increase the robustness of the final prediction results, a stacking-based ensemble algorithm employing a 1D-CNN meta-learner was utilized for fusing the base model prediction outputs. The performance of the models was evaluated by subject-level five-fold cross-validation, while accuracy, precision, recall, F1-score, and AUC were selected as evaluation criteria. The proposed full framework reached the average accuracies of 97.50% ± 2.40%, 99.00% ± 1.90%, and 96.00% ± 2.10% for NC vs. MCI, AD vs. NC, and AD vs. MCI diagnosis, respectively, achieving corresponding AUCs of 0.983, 0.996, and 0.973. Ablation analysis also proved that the patch-based representation, multimodal feature fusion, and stacking helped achieve the desired classification performance, since the full framework significantly outperformed all the ablated configurations ( p < 0.01). Based on the obtained results, it is possible to claim that combining the localized structural and metabolic features provided by the MRI and FDG-PET scans within the framework of the patch-based 3D deep learning algorithm can help effectively discriminate between AD, MCI, and cognitively normal subjects. Additional validation is necessary before applying the framework in clinical practice.