Reinforcement Learning-Based Constrained Optimal Control of Strict-feedback Nonlinear Systems: Application to Autonomous Underwater Vehicles

This paper addresses a constrained neural network (NN)-based optimal tracking scheme for a class of uncertain nonlinear discrete-time systems in strict-feedback form by using a control barrier function (CBF). First, a modified barrier-type cost function is introduced for each subsystem, guiding the actual system trajectory toward the safe set or desired trajectory while avoiding unwanted sets. To address the tracking problem, an augmented system is employed to convert the time-varying optimal tracking to a time-invariant optimal regulation. Then, an actor-critic framework is employed with the backstepping technique to obtain both virtual and actual optimal control policies for each subsystem to avoid the noncausality problem. Additionally, a novel online regularizer method is introduced to reduce catastrophic forgetting in multitasking scenarios by maintaining the significance of weight connections in the critic NN without directly computing the Fisher information matrix (FIM). Further, to guarantee safety during online learning, the actor update law incorporates the safety condition through the utilization of the CBF. Simulation results using underwater vehicles are carried out to verify the effectiveness of the proposed approach.