A statistical journey to DNN, the fourth trip: exploring stochastic gradient descent methods

Stochastic Gradient Descent (SGD) has been fundamental to the success of deep neural networks.This paper provides a comprehensive review of SGD and its extensions.We begin by surveying a wide range of function optimization techniques from numerical analysis and statistics to reveal how modern SGD methods relate to traditional optimization methods.We then explore the characteristics of various SGD-based algorithms to present a systematic comparison of the major SGD optimizers used in deep learning within a unified theoretical framework.