Bridging Faithfulness of Explanations and Deep Reinforcement Learning: A Grad-CAM Analysis of Space Invaders

With the increasing pervasiveness of artificial neural networks in real-world applications, regulations have started enforcing greater transparency in the predictive dynamics of these models.In recent years, research in Explainable AI (XAI) has gained traction, with the goal of enhancing the interpretability of black box models.However, while most of the efforts are aimed at supervised learning on tabular or image data, comparatively less interest has been posed to reinforcement learning (RL).One of the main issues connected to explainability is represented by the potential misalignment between the explanations and the models' true decision-making process, potentially providing, e.g., misleading feature importance outputs.In the present work, we investigate the faithfulness of the explanations generated for an Advantage Actor-Critic model trained to play the Space Invaders Atari 2600 game.We generate feature importance heatmaps by means of Grad-CAM, a popular XAI tool designed for convolutional neural networks with image-based inputs.We then evaluate the faithfulness of these explanations via the Iterative Removal of Features (IROF) metric.While the absence of a baseline for IROF scores complicates result interpretation, our findings indicate that faithfulness estimates exhibit considerable noise.We further observe an overall decrease in faithfulness as the game progresses towards its end.Our study highlights limitations of current XAI techniques in RL contexts and suggests that future work should explore alternative methodologies better suited to RL tasks.