High-Precision and Efficiency Hardware Implementation for GELU via Its Internal Symmetry

The Gaussian Error Linear Unit (GELU), a crucial component of the transformer model, poses a significant challenge for hardware implementation. To address this issue, this paper proposes internal symmetry piecewise approximation (ISPA) and error peak search strategy (EPSS) for high-precision and high-efficiency implementation of the GELU activation function. ISPA only approximates the positive axis of the erf in GELU and then leverages its internal symmetry to calculate the negative axis part. With ISPA, the mean square error (MSE) between the fitted result and the true value can reach 4.29×10−9 with 16 parts of the approximation segment, outperforming the regular method, which achieves 1.19×10−6 with 16 parts. Furthermore, EPSS can automatically find suitable and high-precision intervals for our piecewise approximation method. To evaluate the effectiveness of ISPA and EPSS, we conducted experiments on three different ViT models and observed negligible loss of prediction accuracy. The hardware implementation is on an XCZU9EG FPGA running at 450 MHz. Experimental results indicate that ISPA outperforms existing methods.