Entropy-Guided Head Importance for Token Pruning in Vision Transformers

To address the challenges posed by the high computational load of Vision Transformers, various lightweight methods such as pruning and quantization have been proposed. Among them, token pruning aims to reduce the overall computational load of Vision Transformers by assessing the importance of each token, which is the computational unit, and excluding or reducing computations for tokens with lower importance. To minimize accuracy loss due to token pruning, it is crucial to accurately select the important tokens. In this paper, we propose an entropy guided head importance method for token pruning to achieve this goal.