Explaining model robustness: combining saliency maps and natural robustness testing using XAITK and NRTK

Artificial intelligence (AI) models are often viewed as black boxes, due to the inability to explain how they make their decisions or determine whether they will be robust to new data. As such, model explainability and model robustness are often stated as two key desiderata needed for establishing justified trust in AI, particularly in high-stakes scenarios. Saliency maps, a form of visual explainable AI, can be used to indicate which parts of the input an AI model paid attention to during its decision process. Natural robustness testing can be used to assess an AI model’s performance envelope, by characterizing model degradation across a set of realistic, operationally relevant perturbations. While aspects of model explainability and model robustness have previously been studied independently, we seek to better understand their relationship through quantification of how saliency maps change due to various perturbations. Quantitative changes in the structure and quality of saliency maps may provide a mechanism for understanding model failure modes and edge cases due to various perturbations. Focusing on the computer vision domain, we leverage the open source Explainable AI Toolkit (XAITK) and Natural Robustness Toolkit (NRTK) to perform experiments testing the relationship between saliency map quality and model natural robustness. We demonstrate results across several tasks, datasets, and models, covering various use cases. Finally, we relate this new framework for combined model explainability and robustness to current AI test and evaluation workflows.