On the effect of cross-channel normalization on wide-shallow convolutional networks

Cross-channel Normalization (CN) was first proposed in AlexNet paper as a biologically inspired normalization process mimicking the lateral inhibition phenomenon in biological neurons. However, the effect of such a normalization was not well explored in the literature due to the wide popularity of batch normalization. In this paper, we show that such a type of normalization can significantly enhance the network accuracy when applied to wide-shallow convolutional networks. Wide-shallow networks are more advantageous compared to deep CNNs since they significantly increase parallelism and reduce the sequential computation across layers. Our experiments show that using cross-channel normalization not only improves the accuracy of residual and standard convolutional networks on benchmark datasets but also significantly enhances the performance when considering limited and reduced representation examples.