Efficient token pruning in Vision Transformers using an attention-based Multilayer Network

Vision Transformers (ViTs), although very successful, have a major limitation to overcome, namely the need for significant computational resources to use them. Several approaches have been proposed to limit the resources required to work with ViTs, aiming at pruning the data provided in input to them. In this paper, we propose Token Reduction via an Attention-based Multilayer network (TRAM), the first approach that achieves this goal using a multilayer network-based representation of the attention matrices. TRAM can work with most ViTs without the need for fine-tuning. It makes several contributions to the literature in this research area; in particular, it is characterized by: (i) a new representation of ViTs based on a multilayer network; (ii) a new approach to evaluate the relevance of tokens based on a new centrality measure computed on the multilayer network; and (iii) an approach to reduce the number of tokens based on this centrality measure. We have validated TRAM by comparing it with several state-of-the-art approaches during an extensive experimental campaign carried out on different image datasets. The results obtained demonstrate not only the efficiency but also the effectiveness of TRAM in reducing the computational load of ViTs while still allowing them to provide accurate results. • TRAM represents tokens using an attention-based multilayer network. • TRAM reduces ViT computational demand without requiring fine-tuning. • TRAM improves FPS and GFlops with near-Vanilla model accuracy. • Visual analysis reveals TRAM’s token selection process.