GELU-MSDF: A Hardware Accelerator for Transformer’s GELU Activation Function Using Most Significant Digit First Computation

Transformer models are utilized across diverse fields such as natural language processing (NLP) and computer vision (CV) tasks. Among the activation functions employed in these models, GELU emerges as the most common. GELU is present in each encoder and decoder block, contributing significantly to the transformer’s runtime, accounting for approximately $10 \%$ to $20 \%$. However, implementing GELU in hardware not only requires a substantial area due to the need for module replication to enhance the overall performance of network runtime but also results in high area and power consumption. Therefore, to mitigate these issues, a novel architecture based on Most Significant Digit First (MSDF) has been proposed. In this paper, our proposed GELU-MSDF architecture processes input serially from the Most Significant Digit (MSD) utilizing MSDF arithmetic and a redundant number system. Experimental results demonstrate that GELU-MSDF architecture achieves a reduction in area between 1.75X to 1.84X and power reduction up to 3.2X compared to previous work.