Sparse Group Normalization: An Efficient Normalization Method for Large-Size Feature Map Processing

We proposes a new normalization method, Sparse Group Normalization (SG-Norm), which aims to address the computational efficiency bottleneck faced by deep neural networks when processing large-scale feature maps. SG-Norm introduces learnable channel importance weights, combined with Gumbel-Softmax sampling and Top-K selection mechanisms, dynamically identifies and only utilizes the key subset of each group of channels to calculate the normalization statistics, thereby significantly reducing the computational overhead while maintaining model performance. Through a series of strictly controlled experiments, we prove that SG-Norm can provide up to 2.64 times the speed improvement of standard Group Normalization when processing high-resolution feature maps, while maintaining an acceptable accuracy loss. Experimental results show that SG-Norm with a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{5 0 \%}$</tex> sparsity rate can approach or even slightly outperform GroupNorm in accuracy, and significantly outperform BatchNorm, while providing a significant speed advantage. The 80% sparsity rate can maximize computational efficiency and is suitable for scenarios with extremely high real-time requirements or limited computing resources. This paper elaborates on the design motivation of SG-Norm, the specific implementation principle based on PyTorch, the performance inflection point analysis, and the applicability in different application scenarios, providing a practical and efficient normalization optimization solution for large-size feature map processing.