Evolution-Guided Q-Learning With Dual Swarm Intelligence for Model-Free Optimal Control

In this article, a novel accelerated evolution-guided Q-learning (EGQL) algorithm is introduced to address optimal control problems for unknown nonlinear systems. A novel adaptive evolutionary algorithm is introduced here to enhance problem-solving strategies in two key aspects, which is a concept referred to dual swarm intelligence. First, the novel evolutionary algorithm is incorporated into policy improvement, replacing the gradient descent approach in traditional Q-learning. The method eliminates the need for gradient information in the approximate Q-function, while also enabling more precise policy solutions. Second, the commonly used polynomial model is replaced with a neural network, which is further optimized by the evolutionary algorithm to approximate the critic network. The accuracy of the approximate Q-function is significantly enhanced, improving the precision of value function update. Furthermore, the monotonicity and convergence properties of the algorithm are thoroughly analyzed. Finally, the effectiveness of the EGQL method is validated through two simulation experiments, demonstrating its superiority over traditional Q-learning.