Empirical Performance Comparison of Optimization Algorithms for Machine Learning/Deep Learning

An optimization algorithm is essential for minimizing loss (or objective) functions in machine learning and deep learning. This article compares empirically the performance of 8 optimization algorithms which are used for training in machine learning and deep learning. The optimization algorithms considered in this article consist of 6 well-known algorithms such as SGD, SGD momentum, Nesterov momentum, AdaGrad, RMSprop, and Adam algorithms, and 2 recent algorithms such as KO and mSGD algorithms. Three kind of data is used for the performance comparison of optimization algorithms. First is the use of two functions, which have many local minima. Second is the use of MNIST data set which consists of handwritten digits from 0 to 9. The last one is the use of CIFAR-10 which consists of 10 kinds of images such as airplane, car, cat, dog, and so on. With the three kind of data set, seven cases are considered including FNN, CNN, and AE to compare the performance of 8 optimization algorithms.