TA-ASF: Attention-Sensitive Token Sampling and Fusing for Visual Transformer Models on the Edge

Vision Transformers (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$V$</tex>iTs) have made significant progress in achieving performance comparable to traditional convolutional neural networks in computer vision tasks. However, high computational complexity restricts their application to resource-constrained edge devices. Previous methods for pruning redundant tokens have shown that it is possible to balance performance and computational cost by reducing the number of tokens. Unfortunately, simply removing redundant tokens often leads to the loss of crucial information. To address this issue, we propose a novel token compression scheme called TA-ASF. This scheme considers both the global role of low-importance tokens and the redundancy among similar tokens. TA-ASF employs novel approaches for token sampling and fusion, which are directly applicable to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$V$</tex>iTs without introducing additional trainable parameters. A comprehensive evaluation against several edge devices demonstrates our method effectively reduces model complexity while preserving Top-1 accuracy. Experimental results show that on the ImageNet dataset, the proposed method reduces FLOPs by 37% and increases throughput by 1.48 times on the DeiT-S model, with only a 0.1% decrease in accuracy. Specifically, on the DeiT-B model, the proposed method decreases FLOPs by 35% and increases throughput by 1.52 times while maintaining the same accuracy.