Effective Continuous Equations for Stochastic Adaptive Optimization Algorithms

This work explores the continuous-time dynamics underlying popular adaptive stochastic optimization algorithms used in machine learning, particularly RMSprop and Adam. These methods, which adapt learning rates based on gradient history, are crucial for efficiently minimizing loss functions in large datasets. The study extends the stochastic modified equation framework to describe the behaviour of adaptive algorithms and introduces second-order weak approximations to accurately capture the features of the discrete algorithms in continuous time. Although the noise originates only from batch sampling, adaptive algorithms require additional sources of stochasticity to represent the evolution of internal variables. Furthermore, in the work some scaling rules are obtained, which are essential for deriving the right continuous equations and also offer practical guidance for tuning algorithm parameters effectively.