CHITRA: Clustering Hidden-layer Interpretations through Technical Ranking and Attribution

Developing transparent and reliable AI systems requires a deep understanding of how machine learning models, especially neural networks, make decisions. This project presents CHITRA, a novel interpretability algorithm for neural network hidden layer analysis that combines cosine similarity, Singular Value Decomposition (SVD), and K-means clustering to identify and group relevant neurons. CHITRA’s unique approach involves clustering similar neurons and tracking their activation paths across different inputs to logically deduce their function. Unlike existing techniques such as SHAP and LIME, which struggle with scalability and consistency in bigger and more complicated neural networks, experiments with CNNs and RNNs on MNIST and CIFAR-10 datasets showed that CHITRA surpassed existing techniques like SHAP and LIME in terms of accuracy, precision, and interpretability score, suggesting its potential to provide a more comprehensive understanding of model behavior. To quantitatively evaluate CHITRA’s performance in identifying neuron function, we first established a ’ground truth’ for a small, manually labeled subset of neuron clusters. This involved a detailed, manual analysis of the features each cluster was known to detect. For each neuron cluster, we systematically presented the model with different inputs and recorded the cluster’s activation patterns. For example, if a cluster consistently showed high activation exclusively for images of the digit ’5’, we manually labeled that cluster as a ’5-detector’. This process was repeated for each cluster in the subset, allowing us to create a reliable set of ground-truth labels against which CHITRA’s predictions could be quantitatively compared. We then generated CHITRA’s output for these same clusters and compared its predictions to our ground truth labels. This allowed us to calculate accuracy (the percentage of clusters for which CHITRA’s identified function matched the ground truth), precision (the proportion of CHITRA’s identified functions that were actually correct), and recall (the proportion of all ground-truth functions that CHITRA correctly identified). The results of this analysis demonstrated that CHITRA outperformed existing techniques like SHAP and LIME in these quantitative metrics and qualitative evaluations (like human interpretability assessment). In the past, there have been various instances of poorly trained models that cause misdiagnosis, huge investment losses, and deaths. CHITRA allows researchers to fully understand the entire layout of their models, thereby preventing any of these incidents in the future.