Reinforcement Learning to Stabilize Singularly Perturbed DC-Side Dynamics of Grid-Connected Voltage-Source Converters in Modern AC–DC Grids Using Singular Perturbation Theory and Adaptive Dynamic Programming

The stability and performance of ac–dc systems in grid modernization heavily rely on the rectification mode of grid-connected voltage-source converters (GC-VSCs). Being considered as the heart of the system, its impact is significant. The current-controlled GC-VSC based on the cascade control using a pulsewidth modulation approach is commonly deployed in the smart grid paradigm. This article discusses how the dynamics induced by that type of GC-VSC control structure can be regarded as singularly perturbed systems in modern ac–dc grids. As a result, it proposes a novel optimal control strategy for the voltage control problem with uncertain dynamics using reinforcement learning (RL) via the adaptive (or approximate) dynamic programming method and the singular perturbation theory (SPT). First, by means of SPT, the original optimal control problem is decomposed into two optimal problems with respect to an unknown slow time-scale subsystem and a known fast time-scale subsystem. Second, for the slow subsystem with unmeasurable states, an output-feedback-based off-policy RL algorithm with a guaranteed convergence is given in order to learn the optimal controller in terms of measurement data. Third, a composite controller is established in terms of the obtained fast–slow controllers; its optimality and closed-loop stability are rigorously proved. Unlike the direct full-order design, not only does the proposed decomposition composite design framework bypass the numerical stiffness, but it also alleviates the high dimensionality in the control synthesis. Comparative experiments using testing based on power hardware-in-the-loop simulations and rapid control prototyping methodology reveal the superiority and effectiveness of the proposed method.