A Comparative Study of Optimization Techniques in Deep Learning Using the MNIST Dataset

Objectives: To evaluate the performance of four standard optimization algorithms - Stochastic Gradient Descent, RMSProp, Adam, and AdamW – on the MNIST image classification benchmark using Convolutional Neural Networks (CNNs). It examines the effects of L2 regularization on convergence speed, test accuracy, and generalizability, focusing on the AdamW optimizer. Finally, this study investigates the sensitivity of every optimization algorithm for different learning rates. Methods: CNNs were trained on the MNIST dataset using each of the four optimization algorithms, and experiments were performed. Once the best hyperparameter was found, L2 regularization was used to determine how the performance metrics would be affected, which is one specific focus for AdamW. Different learning rates were tested to test the robustness and adaptability of each optimizer. Findings: The results revealed that AdamW with weight decay outperformed the remaining optimizer-based training regarding test accuracy and validation loss. AdamW had the best generalization performance despite its suitable regularization and SGD and RMSProp being competitive. Moreover, AdamW was more robust over a broader range of learning rates than the other optimization algorithms. Novelty: This work underscores the importance of a lesson that gives more attention to kernel methods in that the role of regularization must not be ignored when describing the generalization performance of deep learning tasks. It analyzes various optimizers against each other at different learning rates and shows that AdamW utilizes L2 regularization to achieve even better performance and robustness. Keywords: Stochastic Gradient Descent, RMSProp, Adam, AdamW, Convolutional Neural Networks