Optimal Secure Control for Cyber-Physical Systems Under False Data Injection Attacks via Incremental Iterative Q-Learning Algorithm

An incremental iterative Q-learning algorithm (IIQLA) is proposed to tackle the optimal secure control problem for cyber-physical systems under false data injection attacks. Within a zero-sum game framework, the secure control problem is transformed into solving an iterative algebraic Riccati equation. To derive the optimal secure control policy from the equation, the IIQLA is developed, which does not require prior knowledge of the system dynamics. This algorithm utilizes two auxiliary variables to separate the behavior policy and the target policy, thereby improving the exploration of data. As a result, the devised control policies are more conservative, leading to solutions that are closer to the optimal policy. Moreover, by introducing an adaptive learning rate, the proposed IIQLA can accelerate the convergence speed and decrease the number of required iterations, thus alleviating the computation burden. The convergence of the proposed IIQLA with different learning rates is also analyzed. In addition, the closed-loop system is guaranteed to be asymptotically stable, and the exploration noise does not introduce bias into the optimal policies. Finally, a simulation example is provided to demonstrate the effectiveness of the proposed approach.