Deep learning optimizer based on metaheuristic algorithms

Deep learning technology has been widely used in artificial intelligence. A successful training of deep learning model is inseparable from the operation of optimization applied by good optimizers. However, the widely used traditional optimizers based on gradient descent, such as stochastic gradient descent (SGD), stochastic gradient descent-momentum (SGD-M), adaptive moment estimation (Adam), and proportional integral derivative (PID) have common problems including being prone to local optima due to gradient vanishing, training termination due to gradient explosion. These problems can be solved by optimizers based on metaheuristic algorithms proposed in the study due to their better global optimization ability and stronger adaptability than gradient descent algorithms. Four different metaheuristic deep learning optimizers were implemented using four different metaheuristics algorithms including particle swarm optimization (PSO), simulated annealing (SA), genetic algorithm (GA), and grey wolf optimization (GWO). These optimizers are tested on the CIFAR and MNIST dataset for image classification task. Experimental results show that these metaheuristic optimizers outperform traditional optimizers.