Analysis Towards Deployment and Acceleration for ViT on a Lightweight RISC- V Processor

Vision Transformer (ViT) is the key architecture for computer vision using deep neural networks (DNNs) such as image recognition, object detection, and anomaly detection. Simultaneously, RISC- V, open-source instruction set architecture (ISA), is the center of attention, and several projects combined DNNs with RISC-V are ongoing. ViT deployment on micro-controller units (MCUs) has high expectations; however, it is a hardness challenge due to the computation cost for ViT. This paper provides a ViT implementation analysis on a small RISC-V processor for efficiency deployments and acceleration. Specif-ically, we target a CV32E40p, a 4-stage in-order 32-bit RISC- V processor core well-known for low-power and small-area. The analysis is performed from the perspective of hardware features (e.g., clock cycles, IPC, the data hazards) for the ViT components, i.e., the Patch embedding, the Layer Normalization, the Attention, and the Fully-Connected Layer. The observation results indicate the decline of instruction execution efficiency in the GEL U activation function and the Attention in addition to overwhelming computation time by matrix-multiplication operation.