CHITRA: Clustering Hidden-layer Interpretations through Technical Ranking and Attribution

Developing transparent and reliable AI systems requires a deep understanding of how machine learning models, especially neural networks, make decisions.This project presents CHI-TRA, a novel interpretability algorithm for neural network hidden layer analysis that combines cosine similarity [8,9], Singular Value Decomposition (SVD) [1, 2], and K-means clustering [3,4,5,6,7] to identify and group relevant neurons.CHITRA's unique approach involves clustering similar neurons and tracking their activation paths across different inputs to logically deduce their function.Unlike existing techniques such as SHAP [13] and LIME [14], which struggle with scalability and consistency in bigger and more complicated neural networks, experiments with CNNs and RNNs on MNIST and CIFAR-10 datasets showed that CHITRA surpassed existing techniques like SHAP and LIME in terms of accuracy, precision, and interpretability score, suggesting its potential to provide a more comprehensive understanding of model behavior.To quantitatively evaluate CHITRA's performance in identifying neuron function, we first established a 'ground truth' for a small, manually-labeled subset of neuron clusters.This involved a detailed, manual analysis of the features each cluster was known to detect.For each neuron cluster, we systematically presented the model with different inputs and recorded the cluster's activation patterns.For example, if a cluster consistently showed high activation exclusively for images of the digit '5', we manually labeled that cluster as a '5-detector'.This process was repeated for each cluster in the subset, allowing us to create a reliable set of ground-truth labels against which CHITRA's predictions could be quantitatively compared.We then generated CHITRA's output for these same clusters and compared its predictions to our ground truth labels.This allowed us to calculate accuracy (the percentage of