Transformer Attention Matrix Multiplication Design using 4 × 4 Systolic Arrays

Transformers have revolutionized natural language processing (NLP) with their multihead attention mechanism, enabling efficient parallel processing of text data. This paper enhances Transformer performance using a systolic array architecture for matrix multiplication. We optimize the computation process by leveraging parallel partitioning to accelerate self and multi-head attention mechanisms within Transformer models. Utilizing the BERT model, we partition its 768-dimensional vectors and process to make multiple smaller 4×4 structures. The experiments were conducted using software simulations and FPGA Zybo 7020 hardware design synthesis. The primary objective is to improve computational throughput and resource efficiency, leading to the development of lightweight and high-performance Transformer models. This study highlights the potential of parallel partitioning with systolic arrays in matrices multiplication advancing the Transformer’s computation processing tasks. Our experiment simulates the parallel partition and synthesizes the hardware architecture design. By comparing the conventional nested loop and parallel partition method, we find a 66.9% speed increase, with maximum frequencies of 91.7. Our findings contribute to optimizing Transformer models for various applications, emphasizing the role of hardware acceleration.