Vision Transformers for Histopathological Image Classification with Efficient Head Pruning

Analyzing histopathological images using state-of-the-art deep learning (DL) models requires specialized approaches due to the unique characteristics of these images. Recently, transformer architectures have been adopted to computer vision, treating images as sequences of patches, similar to words in natural language processing. Taking advantage of this, we employed transformers for histopathological image analysis. We utilized a knowledge distillation (KD) approach in which a student Vision Transformer (ViT) learns from a Convolutional Neural Network (CNN) teacher model. We assessed how the ViT’s attention heads were influenced by the signal received from the teacher. Building on that, this study further investigates the representations learned by the ViT’s attention heads. Our analysis reveals that many attention heads learn similar contexts or concepts, leading to redundancy in the learned representations. Additionally, by analyzing the mean of the maximum attention weights, we observed that only a few heads exhibit high confidence, indicating that they are critical for inference. To optimize the ViT, we performed head pruning at different rates, guided by ”head confidence” and ”head similarities”. Surprisingly, pruning certain heads not only maintained but, in some cases, improved model accuracy while also reducing training time when tested on breast cancer histopathological image classification.