A Flexible Template for Edge Generative AI With High-Accuracy Accelerated Softmax and GELU

Transformer-based generative Artificial Intelligence (GenAI) models achieve remarkable results in a wide range of fields, including natural language processing, computer vision, and audio processing. However, this comes at the cost of increased complexity and the need of sophisticated non-linearities such as softmax and GELU. Even if Transformers are computationally dominated by matrix multiplications (MatMul), these non-linearities can become a performance bottleneck, especially if dedicated hardware is used to accelerate MatMul operators. In this work, we introduce a GenAI BFloat16 Transformer acceleration template based on a heterogeneous tightly-coupled cluster containing 256KiB of shared SRAM, 8 general-purpose RISC-V cores, a 24×8 systolic array MatMul accelerator, and a novel accelerator for Transformer softmax, GELU and SiLU nonlinearities: SoftEx. SoftEx introduces an approximate exponentiation algorithm balancing efficiency (121× speedup over glibc’s implementation) with accuracy (mean relative error of 0.14%). In 12 nm technology, SoftEx occupies 0.039 mm2, only 3.22% of the cluster, which achieves an operating frequency of 1.12 GHz. Compared to optimized software running on the RISC-V cores, SoftEx achieves significant improvements, accelerating softmax and GELU computations by up to 10.8× and 5.11×, respectively, while reducing their energy consumption by up to 10.8× and 5.29×. These enhancements translate into a 1.58× increase in throughput (310 GOPS at 0.8 V) and a 1.42× improvement in energy efficiency (1.34 TOPS/W at 0.55 V) on end-to-end ViT inference workloads.