A Multimodal Transformer with Adaptive GAN for Brain Disease Prediction

Adaptive generative AI is emerging as a significant method to improve brain disease detection by providing innovative methods for brain data augmentation using modality synthesis and handling incomplete and imbalanced datasets. In this paper, we present an adaptive conditional-aware Generative Adversarial Network (GAN) that generates brain modality at a specific stage of disease by creating realistic brain imaging modalities for both Alzheimer’s disease (AD) and brain tumor. Furthermore, it dynamically learns and captures the mapping between different modalities to generate missing brain modalities, such as generating PET from MRI for Alzheimer’s disease detection and CT from MRI for brain tumor detection. Additionally, we combined this adaptive GAN with a new multimodal spatio-temporal transformer model that integrates different brain modalities such as MRI and PET for AD detection and CT with MRI for brain tumor detection and other data types, including genetic and clinical data. This method is enhanced using CNN with an innovative Squeeze-and-Excitation (SE) block. The proposed multimodal transformer encoder incorporates multi-head linear self-attention and a memory-augmented self-attention module to deal with the quadratic complexity of traditional vision transformers. A multi-cross fusion block is designed to extract the brain modalities’ interactions with genetic and clinical data. We evaluate our models using stratified 5-fold cross-validation on three datasets such as the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset for Alzheimer’s disease detection, achieving an average accuracy of 99.05 ± 0.08, and two brain tumor datasets, such as the Cancer Genome Atlas Low Grade Glioma Collection (TCGA-LGG) dataset, where we obtain 98.9 ± 0.12% and 97.50 ± 0.15% accuracy on the brain tumor dataset from Kaggle, respectively. These obtained results demonstrate the effectiveness of our proposed models in both disease prediction and progression tracking, opening a new way for more advanced brain disease detection.