LogSimViT: Logarithmic Similar Pattern Skipping for Hardware Acceleration of ViT

Vision Transformer (ViT) has emerged as a key neural network architecture in computer vision, but its substantial computation and power overheads pose significant challenges, particularly in resource-constrained environments. To address these limitations, we propose Log-similar Pattern Skipping (LPS) algorithm and its accelerator architecture. LPS exploits the redundancy in logarithmically quantized weights to regularly bypass half of FC operations, which accounts for 54% of the total latency of DeiT-Tiny, Small, and Base models on average. Our method and its hardware efficiently accelerate ViT models, achieving a speedup of 1.71× compared to an edge GPU, and 1.70× speedup and 1.33× higher energy efficiency compared to state-of-the-art ViT accelerators, with minimal accuracy loss.