Automatic Adaption of the Step Size in Gradient Descent Training

Selecting an appropriate step size is critical in Gradient Descent algorithms used to train Neural Networks for Deep Learning tasks. A small value of the step size leads to slow convergence, while a large value can lead to instability. In this paper, we present ACME, a gradient descent algorithm that automatically updates the step size to its optimum value. When combined with Adaptive Moment Estimation (ADAM) and/or Adaptive Sharpness Aware Minimization (ASAM), ACME achieves performances similar to them, but without the burden of carefully adjusting the learning rate or designing intricate learning rate scheduling policies.The code of ACME, ready to be used by itself or in combination with ADAM and/or ASAM, is available at https://github.com/ignasi00/acme.git.