Learning Rate Optimization in Neural Networks: Challenges and Solutions in Training Dynamics

One of the most difficult problems in deep learning is finding the appropriate learning rates for learning the neural network. In this comprehensive analysis, we investigate the basic questions related to learning rate choice, their impact on training dynamics, as well as the current approaches. To develop good, robust, efficient training procedures for modern neural networks, one has to understand and effectively manage learning rates.