A Layer-wise N:M Sparsity Aware Transformer Accelerator leveraging Temporal Locality with Butterfly Network

In recent years, Transformer-based neural networks have driven significant advancements across various fields. However, the attention mechanism results in high memory access and computational demands, making deployment on edge devices challenging. To address these issues, we propose a layer-wise N:M sparsity-aware Transformer accelerator with two key innovations: 1) a layer-wise N:M sparsity-based parallel dimension compression strategy that effectively leverages the temporal locality of feature values, and 2) a butterfly network-based scattering unit that ensures correct accumulation of partial products after multiplication. Experimental results demonstrate that the proposed accelerator achieves up to a 3.86× improvement in runtime cycles. When running Deit-s on Imagenet-1k dataset, the accelerator delivers an energy efficiency of up to 34.4 TOPS/W at 68.9% sparsity under an industrial 28-nm technology, with only a 0.39% accuracy loss, representing a 1.46~1.48× improvement over prior work.