How Explainable Are Black-Box Models in Brain Tumor Classification

Adopting AI models in the healthcare system is often impeded by concerns over the lack of transparency and interpretability. In our study, we address this challenge of black box nature and explain it through explainable AI (XAI) techniques— SHAP, GRAD-CAM, and LIME. We employed pre-trained VGG16 and VGG19, which are considered black box models, and used the latter XAI techniques to explain the following model’s black box nature by identifying the critical pixels or superpixels in brain MRI images that significantly influence the model’s decision-making. Our analysis featured the XAI algorithms’ behavior when applied to the models and how the different behavior of the XAI helps enhance model interpretability, particularly investigating how the extra convolution layer in the VGG network affects the behavior or effectiveness of the explainable AI algorithms. Also, we address higher performance metrics, such as accuracy, which do not necessarily equate to the black box model’s ability to correctly identify relevant pixels in the classification. The ascendancy of VGG19 over VGG16 in terms of zeroing in on the critical pixels and the effectiveness of the XAI techniques are further explained through counterfactual explanation. Analysis of the highlights and counterfactual explanations among the XAI techniques presents SHAP as the most effective XAI.