Koopman-Based Reinforcement Learning for LQ Control Gains Estimation of Quadrotors

In this research, Koopman operator theory is employed to achieve faster training time and improved performance of a reinforcement learning (RL) based linear quadratic controller (LQ). The proposed methodology, called K-RLLQ, is implemented for the trajectory tracking problem of a quadrotor UAV. Using the evolution of analytically derived Koopman generalized eigenfunctions allows for the embedding of quadrotor nonlinear dynamics into a quasi-linear model. Specifically, the resulting Koopman based quadrotor dynamics has linear state matrix and state dependent control matrix. Additionally, the obtained formulation is fully actuated, hence, compared to traditional model based hierarchical control the advantages are twofold: i) the controller can be formulated using linear control strategies in Koopman formulation which will result in a nonlinear control law in the original state space; ii) the trajectory tracking task can be achieved through a single control loop. Using this formulation, an RL agent is trained to estimate the controller parameters of a linear quadratic control law. Notably, it is shown that, using a reward function and observation space based on Koopman generalized eigenfunctions over the state space, leads to a considerably faster training time and improved overall performances.