Asynchronous Iterative Q-Learning: Optimal Control for Nonlinear Discrete-Time Systems

This paper introduces a novel asynchronous iterative Q-learning algorithm (AIQL) to solve the optimal control problem for nonlinear discrete-time systems. AIQL utilizes two Q-value functions, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{Q}_{i}$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{Q}_{i-n}$</tex>, where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{Q}_{i}$</tex> is used for improving the control policy, while <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{Q}_{i-n}$</tex> is used to evaluate the policy's performance. The advantage of this structure is that it decouples policy evaluation from policy improvement, thus reducing estimation bias. Additionally, the paper provides a convergence proof for AIQL, showing that both <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{Q}_{i}$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{Q}_{i-n}$</tex> converge to the optimal solution. Neural networks based on the actor-critic framework are employed to implement asynchronous iterative Q-learning. Simulation results demonstrate that asynchronous iterative Q-learning outperforms the traditional iterative Q-learning (IQL) algorithm, particularly in terms of reducing estimation bias and achieving better control performance.