Linear Quadratic Control of Unknown Nonlinear Systems Using Model-Free Reinforcement Learning

This article proposes linear quadratic controllers for unknown nonlinear systems with noise. Scenarios with unknown underlying nonlinear dynamics but measurable system states are considered. The unknown nonlinear systems are lifted to higher dimensional model-free linear predictors, with approximately linear system evolution by adopting the Koopman theory. The linear quadratic control (LQC) problem of the original nonlinear system is transformed into an LQC problem with a model-free linear predictor. To determine the gain of an optimal feedback control policy, quadratic cost, and associated value functions are designed for the lifted state, enhancing the theoretical performance of model-free reinforcement learning algorithms through control theory. An improved average Q-learning algorithm is presented to learn the gain of the optimal control policy based on the lifted noisy state observations. Helicopter experiments are presented to demonstrate the effectiveness of the model-free approach over representative methods.