An Agile Systolic Array-Based Hardware Accelerator for Scalable Multi-Head Self-Attention

As attention mechanisms find increasing application in computer vision, numerous neural networks are now integrating convolutional neural networks (CNNs) with attention layers to enhance performance. As a typical representative, the Bottleneck Transformer (BoT) network effectively balances the extraction of local features and modeling of global context, leading to significant performance improvements. However, the multi-head self-attention (MHSA) layer of BoT deviates from conventional MHSA by incorporating a wider range of matrix operations, posing challenges in computational complexity and memory bandwidth. This letter introduces a customized accelerator for the MHSA layer of BoT. The accelerator employs a configurable systolic array architecture, designed to support various matrix operations, including matrix transposition, addition, multiplication, and a hardware-optimized Softmax, facilitating hybrid scalable MHSA layers. Implemented on an FPGA, the proposed accelerator achieves a performance of 143.98 GOPS at a frequency of 320 MHz. When compared to CPU implementations, our design offers a 7.18× reduction in inference time.