XAINT: eXplainable AI through Iterative Network Truncation

In recent years, advancements in Explainable AI (XAI) have leveraged game-theoretical approaches, notably SHAP (Shapley Additive Explanations), to address the opacity of complex machine learning models. These methods offer insights into model behavior by distributing predictive importance among features, based on cooperative game theory. Despite this progress, developers still struggle to interpret how specific layers of neural networks contribute to decision-making. To address this, we introduce XAINT (eXplainable AI through Iterative Network Truncation), a novel visualization method that reveals SHAP values across all the model subnets. By iteratively truncating the network and visualizing the impact of each layer, XAINT provides developers with a deeper understanding of how different layers influence model predictions. Our experiments revealed that the hidden representations hold significant information—often sufficient for effective classification—as the distribution of SHAP values in intermediate layers closely resembles those in the final layers. XAINT ultimate goal is to optimize networks by identifying and removing non-essential layers.