An intelligent Gradient Method using Particle Swarm Optimization

The Gradient algorithm is the simplest and most widely used method in optimization problems and machine learning.The convergence rate of this method strongly depends on choosing the suitable value for the step size.Choosing a very small value can cause a low convergence rate, on the other hand, choosing a very large value may also cause divergence and oscillation around the optimal point.Usually, the step size is chosen larger in the initial steps of optimization and as the execution steps progress and the optimal solution is approached, its value decreases.The optimal setting of this hyper-parameter is experimentally and by trial and error and has to be done separately for every problem, so it takes a lot of time.On the other hand, in swarm intelligence optimization methods, including the particle swarm optimization (PSO) algorithm, the step size (the length of movement) is automatically adjusted during the execution of the optimization method.Also, in these methods, few parameters need to be tuned and a predetermined range is available for this purpose.In this paper, by combining the PSO and Gradient method, an intelligent optimization method based on the gradient is presented, which does not need to adjust the step size.The performance of the proposed gradient algorithm is evaluated on ten benchmark functions and it is observed that the proposed algorithm has a better convergence rate than the Classic Gradient algorithm in reaching the optimal solution.