An explainable hybrid CNN–vision transformer framework for brain tumor detection and classification using MRI

Despite the significant strides towards exploiting deep learning in medical image analysis for diagnostic purposes, the diagnosis of brain tumors is still a challenging task where the localization, multi-class classification and explainability of brain tumors are not fully included in a single deep learning model. Current methods treat both tumor and non-tumor areas equally, potentially contributing to an underrepresentation of discriminative information and the ideas of explainability are typically used as a post hoc visualization method. To tackle these problems, this work proposes an Explainable Artificial Intelligence-based Brain Tumor Network model (XAI-BTNet), which is a segmentation-guided hybrid architecture by combining CNN and Vision Transformer (ViT) with EAI. A CNN–UNet model trained with BraTS 2021 data is designed to generate tumour masks to reduce the background noise patch tokens before the self-attention stage of ViT, allowing the model to learn the tumour-relevant features. In addition, a consistency mechanism based on Grad-CAM is embedded in the training process for the consistency of the attention maps generated with the tumors areas, in order to improve interpretability. Thereafter, the proposed framework was tested on the MRI dataset from the Figshare database for multi-class brain tumour detection and validation of the segmentation results using the Brain Tumor Segmentation (BraTS) 2021 dataset. Experimental results show that XAI-BTNet has a classification accuracy of 98.6%, a sensitivity of 97.8% and a specificity of 98.9%. These insights reveal the capability of XAI-BTNet in accurate and explainable brain tumor classification and its potential applications in computer-aided diagnosis and clinical decision support.