SAND: Enhancing Open-Set Neuron Descriptions through Spatial Awareness

We propose Spatially-Aware open-set Network Dissection (SAND), a technique to identify and label the learned representation of the neurons of deep vision networks. Con-trary to earlier open-vocabulary neuron explanation meth-ods, we also leverage a neuron's spatial pattern of activation to guide our predictions towards more accurate and rel-evant concepts, while avoiding being misled by confounding visual information. We highlight important regions for a neuron through image masking, which has the advantage of being able to block out irrelevant concepts from an im-age, handling irregularly shaped activation regions, and re-vealing the visual concepts that a neuron learns in order to identify objects. We use CLIP to connect highly activating image regions with descriptive concepts, and measure the quality of our results through human evaluation. Further, since such manual evaluation can be highly time consuming, costly, and unscalable, we also propose an automated approach which uses image generation to get quantitative feedback on the generated concepts. Finally, as an application of our interpretability method, we demonstrate how it can be tuned to the medical domain. Our code is available at https://github.com/Trustworthy-ML-LabISAND