Hardware Accelerator for Deep Neural Networks on Edge Devices Based on RISC-V Open Standard Instruction Set Architecture

Recent advances in transformer neural network architecture are constrained by their substantial computational demands, which pose significant challenges in edge computing environments. In these limited network connectivity settings, local data processing on resource-limited embedded processors becomes necessary. We designed a hardware custom function unit (CFU) to accelerate neural network computation in edge computing environments. Our circuit functions as a co-processor to an embedded processor and implements new custom instructions compatible with the Reduced Instruction Set Computer <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$5^{\mathrm{t}\mathrm{h}}$</tex> Generation (RISC- V) Open Standard Instruction Set Architecture (ISA). We implemented a systolic array of programmable processing elements (PE) with local cache to offload computation of dense matrix multiplication from the embedded processor. Through simulation, we achieved upto threefold improvement in processing speed for matrix multiply-and-accumulate operation of transformer models with three encoder layers and 256 dimensions. Our design provides a pathway to seamlessly integrate hardware acceleration to enhance inference computation in embedded processors by leveraging on the advantages of open standard ISA. With defined interfaces to a RISC- V processor core, our hardware can be imported as an IP block in programmable or application specific design flows. On the software side, dense matrix multiplications instructions are replaced with our custom instructions while the rest of the instruction compilation and assembly steps can continue to leverage on open-source software tools that convert Python ONNX models to RISC-V instructions.