Low-Precision Normalization Algorithm and Accelerator for Neural Network Training

As one of the most essential techniques in modern deep learning, normalization layer largely improves the convergence speed and performance of deep neural networks (DNN). However, calculating normalization statistics during training is costly, as the two-pass algorithm requires repeated accumulation on the same data. Although parallelized one-pass algorithms are used for variance calculation, low-precision floating-point arithmetic often leads to catastrophic cancellation and accuracy degradation due to uncentered batch statistics and rounding errors. In this paper, we propose Welford-Pairwise Normalization (WPN), a novel normalization algorithm with accelerator design for low-precision training. WPN resolves numerical instability while performing parallel computation. Implemented on the Xilinx Alveo U280 FPGA, WPN achieves up to 28× throughput improvement compared to standard normalization layer with less than 1% accuracy loss with BEiT3 vision transformer, ResNet, MobileNet, and VGG16 model.