Efficient Feature Extraction For Vision Transformer Model Using a Custom CNN Accelerator

As the application of Vision Transformers (ViT) in malware detection has increased significantly over the past few years. Recently, it has become clear that these models need large datasets for training and substantial computation time. Lately, certain researchers have directed their attention toward enhancing the training efficiency and computation time of ViTs by utilizing neural network-based accelerators. However, current models lack the necessary optimization strategies and hardware resources to run the Vision Transformer efficiently on FPGAs while maintaining model accuracy. In this study, we aim to present a dedicated accelerator on a Genesys 2 FPGA board. The accelerator handles input images, carrying out advanced feature extraction through convolutional neural network (CNN) techniques. The accelerator seamlessly employs a systolic array as its main computation unit and is integrated with the AS1061 processor through the AXI4 communication protocol. Leveraging the AS1061’s high computational power, memory bandwidth, and support for various RISC-V extensions, our approach ensures peak system efficiency. The accelerator was operated at 200 MHz with a peak performance of 116 GOP/s and minimal on-chip power consumption of 2.498 W. Lastly, when the ViT inference model, along with the accelerator, was executed on the FPGA, we achieved an overall reduction of 2.8x in inference latency when compared to pure CPU execution.