Edge-BiT: Software-Hardware Co-design for Optimizing Binarized Transformer Networks Inference on Edge FPGA

Binarized Transformer (BiT) networks have emerged as a notable advancement in artificial intelligence, reducing computational complexity by replacing traditional multiply-accumulate operations with simpler XNOR-popcount operations. This paper proposes Edge-BiT, a framework that employs software-hardware co-design to optimize BiT networks inference on edge FPGAs. We design a training method tailored for hardware acceleration, including the Exponential-Sparse Data Format (ESDF). This format is not only compatible with a broad range of existing BiT networks but also enhances their accuracy while reducing inference time on hardware platforms. Additionally, we develope a Compiler and an Instruction Set Architecture (ISA) based on a trigger-update mechanism to support inference across various BiT networks. On the hardware side, Edge-BiT features a unique Sparse Matrix XNOR Unit (SMXNOR) and an optimized Post Process Unit (PPU), both specifically tailored for efficient binary operations. Our empirical evaluations show that our training strategy, coupled with ESDF, increases the accuracy of BiT tasks by 4.6--34.1%. Edge-BiT significantly outperforms traditional CPU and GPU solutions, achieving speedups of up to 23.54× and 9.34×, respectively, and demonstrating up to 424.75× and 121.91× greater energy efficiency. Compared to existing FPGA accelerators for Transformer networks, Edge-BiT exhibits superior performance while utilizing fewer resources, achieving latency reductions of 1.16--2.78 × and improving energy efficiency by 1.34--7.43 ×. To the best of our knowledge, Edge-BiT is the first study on a software-hardware co-design framework for various BiT networks that achieves both high accuracy and excellent performance.