A Comparative Analysis of Optimization Methods for Classification on Various Datasets

Optimization studies how a variety of mathematical structures can be analyzed through the minimization or maximization of a function. In deep learning (DL), optimization encompasses everything from hyperparameter tuning to weight and bias adjustment until the convergence of a loss or cost function (J), such that the model's performance metrics and reliability in classification and regression are increased. Over the last few years, stochastic gradient methods and their variants—or Adaptive Gradient Methods—have become very popular, with varying levels of success or otherwise. This study provides a neat comparison of adaptive gradient methods with respect to their accuracies and Cross-Entropy Loss (CEL) in the mentioned tasks; it tested nine optimization algorithms across three CNN architectures on MNIST, Fashion-MNIST, and CIFAR-10 datasets over 30 epochs. The overall top-performing optimizers were SGD, RMSProp, Adam, and Nadam, whereas Adagrad and Adadelta consistently performed lower.