Interpretable correlation map (ICM): a derivative of the feature maps to improve visual explainability in convolutional neural networks

Convolutional neural network (CNN) models play major roles in artificial intelligence (AI) systems. Hence, it is important to study them further and improve their visual explainability to develop explainable AI (XAI). The CNN models make predictions using the spatial information that are generated through a convolutional process and hidden in the feature maps of their final convolutional layers. However, the problem is that the final convolutional layer only provides the semantic meanings which cannot be easily interpreted by human intelligence to understand the connection between the input features and the model predictions. Thus, it makes the explainability difficult. This paper proposes a technique, called the interpretable correlation map (ICM) framework, to study how CNN models prioritize the features to make predictions. The ICM framework clusters the feature maps into interpretable correlation maps by extracting the correlation properties of the hidden semantic meanings. We have studied the feature maps of the pretrained models, VGG16, VGG19, ResNet50, and InceptionV3, along with a plant leaf disease dataset, to develop and validate the ICM framework. We have also used the Grad-CAM visual explainability tool, as benchmark, to validate the proposed ICM framework. For clustering the feature maps with respect to their correlation properties, we have used the well-known k-means clustering technique with k=16. Our finding is that the ICMs that are extracted from the feature maps of the CNN models allow us to identify the subspaces of the feature maps, selected by the models, to make predictions. The ICM framework also aggregates these relevant subspaces of the feature maps to localize explainability. Simulations have shown that these localized feature maps are comparable to the related Grad-CAM heatmaps. Hence, our conclusion is that the proposed ICM framework can act as a tool to improve CNN models for developing trustworthy XAI systems.