Matrix Accelerator Designed for Vision Transformer

Due to the popularity of large language models such as Generative Pre-Trained Transformer (GPT), many researchers have invested in the research of the Transformer model. Vision Transformer has become one of the most popular research directions in the field of computer vision. Different from the previous Convolutional Neural Network, Vision Transformer blocks the image and adopts Self-Attention. The model can capture the information in the image globally, not just local features. However, a large number of matrix operations consume a lot of GPU resources and memory space. The goal of this paper is to design a dedicated chip to support the digital storage and processing of images. The goal is to design based on the Vision Transformer architecture, focusing on the implementation of matrix multiplication and Softmax of Attention in the Encoder. The completed circuit is simulated running at a clock frequency of 100MHz. Compared with CPU hardware acceleration, the running speed is 51.5 times faster.