Understanding the benefits of noise in deep neural networks via the information bottleneck

Abstract From the perspective of information transmission and visualization in deep neural networks, the information bottleneck theory offers a method to measure the learning of representations within networks. This thereby provides a valuable framework for understanding the benefits of noise in deep neural networks. In this paper, we derive a noise-boosted activation function based on the suprathreshold stochastic resonance model and analyze its information transmission characteristics. Our findings demonstrate that the mutual information between input and output of the noise-boosted activation function exhibits a resonant behavior, characterized by compression followed by representation as the noise scale parameter increases. This noise-boosted activation function embeds the noise scale parameter within the hidden layers of neural networks, which makes adaptive adjustment possible in stochastic gradient descent methods. Subsequently, comparative experiments of fully connected or convolutional neural networks established on the noise-boosted activation function show that the noise scale in the hidden layers plays a crucial role in accelerating early-stage convergence and improves overall network performance and representation capacity. These effects are visualized layer by layer on the information plane, providing a deeper understanding of the benefits of noise in deep neural networks.