A 77.79 GOPs/W Retentive Network FPGA Inference Accelerator with Optimized Workload

Retention Network (RetNet) is a novel neural network model, with a inference complexity of O(1), considered to be the successor of the Transformer model. This work presents an inference accelerator supporting RetNet and DNN algorithms focus on dataflow optimizing, model quantization for improved parallelism and reusability, and addressing K-V cache issues in Transformers. To meet model inference storage bandwidth requirements, we used output block stationary (OBS) dataflow to explore data reusability in time and space. A hardware accelerator for the position encoding module is designed, reducing storage bandwidth and on-chip storage during Extrapolatable Position Embedding (XPOS). Compared to Transformer models of the same size architecture, this work has better inference accuracy, achieving energy efficiency ratio of 77.79 GOPs/W, a 12.5x speedup, and 2.13x energy efficiency compared to GPU implementations, which is 1.33x better than the transformer work baseline. Meanwhile, the inference process of Muti-Scale-Retention(MSR) is completed with the smallest on-chip RAM of 350KB, thereby enabling a more efficient ASIC implementation of hardware accelerators.