Comparative Analysis of Optimization Algorithms

AbstractThe performance of four optimization algorithms—SGD, SGD with Momentum,Adam, RMSprop, and Adagrad—was compared in this study on resource-constrainedfeedforward neural networks. The analysis by convergence speed, escape successfrom local minima, which was set to ±0.01 perturbations, and ETR (Escape-TimeRatio) which was proposed as a new method of linking recovery efficiency to trainingtime. In the experiments, the same network architectures and standardized datasets(MNIST, CIFAR-10, Iris, and Wine) were used. The results indicated that Adamhad the highest ETR and accuracy, while SGD with Momentum was the most bal-anced regarding speed, stability, and energy efficiency. The outcomes of the studygive practical recommendations for the choice of optimizers in edge and low-powerAI systems and also help to establish the practice sustainable, energy-aware neuralnetwork training.