Transformer and Convolutional Neural Network: A Hybrid Model for Multimodal Data in Multiclass Classification of Alzheimer’s Disease

Alzheimer’s disease (AD) is a form of dementia that progressively impairs a person’s mental abilities. Current classification methods for the six AD stages perform poorly in multiclass classification and are computationally expensive, which hinders their clinical use. An efficient, low-computational model for accurate multiclass classification across all AD stages is needed that can integrate both local and global feature extraction. This study uses rs-fMRI, clinical data, and transformer-based models to classify six AD stages. The proposed network is a hybrid of two architectures, namely a transformer and a convolutional neural network (CNN). The model addresses multiclass classification by examining the brain’s functional connectivity networks based on rs-fMRI data and clinical data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). The proposed architecture leverages CNNs for local feature extraction and transformers for global context; this method employs the contextual attention power of transformers to improve the multiclass classification accuracy of AD. The k-fold cross-validation method was employed to evaluate the performance of the proposed model. For the multiclass classification of six stages, the average accuracy of the model was 96%. For binary classification, accuracies were 98.96% (AD vs. MCI), 99.65% (AD vs. CN), 98.44% (AD vs. LMCI), 96.88% (AD vs. EMCI), and 98.36% (AD vs. SMC). These results highlight the potential of the proposed network in achieving high accuracy for both binary and multistage AD classification with limited computational resources. The proposed method was also compared to benchmark algorithms and outperformed them; it was substantially less computationally expensive while maintaining its accuracy.