A New Adaptive Dynamic Programming Reinforcement Learning Method for Discrete-Time Linear-Quadratic Optimal Control Problems

Abstract In this article, we present a new policy iteration approach to solve the linear-quadratic (LQ) optimal control problem for infinite horizon discrete-time dynamic systems under the assumption that the system input matrix is known and the state variables are accessible. The adaptive dynamic programming (ADP) methodology has been applied and widely used in many areas on reinforcement learning of corresponding continuous-time problems. However, the built-in constraint has limited its generalization to discrete-time problems as the prior knowledge of the system matrix is required. With our newly proposed method, the system state matrix can be recovered from the iterative reinforcement learning procedure so that the newly developed partial model-free policy iteration approach can also serve as a linear system identification technique. A realistic data-driven example, based on an online tracking and planning navigation mathematical model of a nonholonomic tracked vehicle under a leader–follower formation, is included to demonstrate the efficiency and robustness of the newly proposed methodology.