On the Performance of Stochastic Gradient Methods with Momentum in Time-Varying Regimes

Abstract We explore how Stochastic Gradient methods with momentum perform within a time-varying framework by establishing bounds on their tracking errors, specifically focusing on quadratic cases. Notably, we find that momentum methods achieve, in high drift-to-noise regimes , i.e., when the rate of change of the dynamic optimum prevails on the variance of the gradient noise, smaller neighborhood of convergence compared to stochastic gradient descent. To the best of our knowledge, this is the first proof that a momentum method can improve upon stochastic gradient descent’s tracking error bounds in a time-varying setting. We then investigate, for a given learning rate, the optimal choice of the momentum parameter that minimizes the tracking error bound.