Vision-Language Model for Early Alzheimer’s Diagnosis by Integrating MRI and Clinical Reports

Alzheimer’s disease (AD) is a progressive neurodegenerative disorder that impairs cognitive function and eventually leads to severe disability. Early detection of AD is crucial for timely intervention and effective treatment planning. Automated diagnostic methods using Deep Learning (DL) models have been made possible owing to rapid advances in artificial intelligence (AI) in recent years. Traditional unimodal imaging often fails to capture complex relationships between brain imaging and textual clinical evaluations, limiting diagnostic accuracy. Therefore, this study proposes a vision language model (VLM) that overcomes these challenges by integrating both modalities (MRI and clinical text data). Our proposed approach utilizes a Swin transformer as the vision model to extract structural features from the MRI images, while the textual descriptions of the clinical reports are being processed using BioBERT. The proposed approach combines these modalities through multimodal learning based on CLIP / Blip, allowing the model to identify correlations from imaging patterns to language biomarkers. Our proposed model is trained and evaluated on the real-world dataset known as the AD Neuroimaging Initiative (ADNI). Experiments show that the proposed multimodal approach outperforms unimodal models by achieving a classification accuracy of $\mathbf{9 2. 5 \%}$ and an AUC-ROC of 0.97%. This work demonstrates the power of using vision language models to diagnose neurodegenerative diseases. It lays the foundation for potential AI-enabled, clinically interpretable solutions for detecting AD in a well-disciplined manner.