The Role of the Learning Rate in Layered Neural Networks with ReLU Activation Function

Using the statistical physics framework, we study the online learning dynamics in a particular case of shallow feed-forward neural networks with ReLU activation.By expanding the activation function in terms of Hermite polynomials we derive analytical results for the evolution of order parameters for any learning rate.Moreover, we compare our results with online gradient descent simulations and show how our method describes the typical learning curves.We also present results on how the learning rate affects the overall behavior of the network and its equilibria, showing different learning regimes and critical values of the learning rate.