Pseudoinverse Learning Algorithm As a Unified Framework For Normalization In Deep Learning

In recent years, several normalization methods have been proposed in order to train neural networks, including batch normalization, layer normalization, weight normalization, and group normalization. However, it does not exist a unified framework which can unify these normalization techniques for easy using to most researchers. In this paper, we proved that most of the normalization methods can be realized in a unified framework, namely PseudoInverse Learning (PIL) algorithm. When training neural network with PIL algorithm, no other normalization methods are needed further since normalization is natural embedded in PIL algorithm. And like normalization techniques, PIL algorithm not only can speed up training deep neural networks, but also can act as a regularizer to improve generalization when utilizing low rank constraint PIL for autoencoders. Hence, our proposed unified framework can help researchers to tackle the difficult of choosing proper normalization methods, and this can greatly facilitate their research process.