Famous: Flexible Accelerator for the Attention Mechanism of Transformer on Ultrascale+ FPGAs

This paper proposes FAMOUS, a flexible hardware accelerator for dense multi-head attention (MHA) computation of Transformer neural networks (TNNs) on field-programmable gate arrays (FPGAs). It is optimized for high utilization of processing elements and on-chip memories to improve parallelism and reduce latency. An efficient tiling of large matrices has been employed to distribute memory and computing resources across different modules on various FPGA platforms. The design is evaluated on Xilinx Alveo U55C data center cards containing Ultrascale+ FPGAs. Experimental results showed that it can attain a maximum throughput, the number of parallel attention heads, embedding dimension, and tile size of 328 (giga operations/second (GOPS)), 8, 768 and 64 respectively on the U55C. Furthermore, it is <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.28 \times$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.6 \times$</tex> faster than the Intel Xeon Gold 5220R CPU and NVIDIA V100 GPU respectively. It is also <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\text{1. 3} \times$</tex> faster than the fastest state-of-the-art FPGA-based accelerator.