A Streaming Transformer Accelerator with Efficient On-Chip Normalization

Transformer has achieved state-of-the-art results in many fields. However, the extensive and dynamic matrix multiplications and normalization operations (Softmax and Layer Normalization (LN)) result in a significant amount of data movement and computation. In this work, a software-architecture co-design method that can effectively addresses the challenges of Transformer inference is proposed. The architecture part implements a low-cost normalization by introducing a fast square root method to enhance the efficiency of LN. Additionally, a hardware-friendly Softmax has been deployed to further reduce the normalization overhead. Benefiting from the low-cost normalization, a streaming transformer inference dataflow has been designed to reduce memory access. The software part ensures the inference accuracy when implementing the proposed low-cost normalization through a two-step fine-tuning method. The proposed accelerator achieves $8.86 \times$ and $6.24 \times$ energy efficiency improvements compared to state-of-the-art attention accelerators ${\mathrm {A}}^{3}$ and SpAtten.