A Transformer-Based Multi-Modal Deep Learning Approach for Brain Tumor Classification with Explainable AI

This paper investigates the applications of explainable AI (XAI) in Brain tumors, A complex medical conditions that require accurate and early identification for effective treatment. Magnetic Resonance Imaging (MRI) plays a key role in their diagnosis due to its high spatial resolution. However, manual interpretation of MRI scans is often time-consuming and subject to variability. Artificial Intelligence (AI), especially in the form of automated image analysis, has opened up new possibilities to improve diagnostic accuracy and provide tools that help radiologists reduce their workload and increase diagnostic consistency. To solve the brain tumor problems, this study introduces the ResNet50 model, modified for brain MRI. Using a dataset of 2212 valid MRI images, the model achieved 97.35% accuracy with good classification and mixed data. Its power across a variety of imaging scenarios truly demonstrates its potential as a clinical decision-support tool. By providing second opinions and reducing misclassification, it helps reduce human diagnostic errors and increases the confidence in AI-assisted diagnosis. Deep Learning architectures, especially CNN-based models like VGG, Alex Net, and ResNet have made a significant impact medical imaging by extracting meaningful features from scans. While these architectures provide good segmentation and region accuracy, traditional models suffer from limitations when modeling large contextual regions in 3D images. Recent developments such as Swin UNETR utilize transformer-based encoders to effectively learn long distance relationships and multiple features, establishing a new standard in tumor segmentation tasks, especially when compared to competing approaches such as BraTS. Despite progress, many existing models rely solely on a single imaging modality, limiting their ability to capture the full complexity of brain tumors. Furthermore, model interpretability remains a pressing issue; without transparency, AI systems face skepticism in clinical use. Current methods often function as opaque decision engines, which can hinder trust and clinical adoption. Bridging the gap between high performance and interpretability is necessary, particularly in environments where diagnostic certainty is paramount. To overcome these limitations, a novel multi-modal framework is needed that combines diverse data types (e.g., MRI sequences, clinical data) with transformer architectures and Explainable AI (XAI) techniques. By embedding explainability into transformer-based models, tool like SHAP, LIME, and Grad-CAM can clarify how decisions are made, fostering transparency and user confidence. This integration represents a promising direction for brain tumor diagnostics, where models must not only be accurate but also interpretable, generalizable, and trusted in real-world healthcare settings. This study focuses on the development of a robust Brain Tumor Detection system employing a Deep Learning (DL) approach, primarily utilizing Magnetic Resonance Imaging (MRI) data. The methodology involves a comprehensive pipeline encompassing preprocessing, segmentation, and feature extraction techniques to enhance the quality of input data for subsequent analysis. Various machine learning algorithms, with a particular emphasis on the Support Vector Machine (SVM), are employed for classification tasks to distinguish between normal and abnormal brain tissues. Performance metrics such as Root Mean Square Error (RMSE), recall, sensitivity, precision, F-score, specificity, Probability of Misclassification Error (PME), and accuracy are used to assess the model's efficacy. The integration of these metrics allows for thorough evaluations of the detection system, ensuring high classification precision, sensitivity to abnormalities, and overall accuracy in classifying brain tumor cases. This study also discusses the development of the advanced medical diagnosis system through the fusion of advanced imaging technologies and deep learning methodologies for improved. The proposed approach contributes to the advancement of medical diagnostics through the fusion of advanced imaging technologies and deep learning methodologies for improved brain tumor detection and classification.