Offline to online reinforcement learning for optimizing FACTS setpoints

With the growing electrification and integration of renewables, network operators face unprecedented challenges. Coordinated control of Flexible AC Transmission Systems (FACTS) setpoints using real-time optimization techniques has been proposed to substantially improve voltage and power flow control. However, optimizing the setpoints of several FACTS devices is rarely done in practice. In part, this can be derived from the challenges with model-based methods. As alternative control methods, data-driven methods based on reinforcement learning (RL) have shown great promise. However, RL has its own challenges that include data and safety during learning. Motivated by the increasing collection of data, we study an RL-based optimization of FACTS setpoints and how datasets can be leveraged for pre-training to improve safety. We demonstrate on the IEEE 14-bus and IEEE 57-bus systems that an offline to online RL algorithm can significantly reduce voltage deviations and constraint violations. The performance is compared against an RL agent learning from scratch and the original control policy that generated the dataset. Moreover, our analysis shows that dataset coverage and the amount of pre-training updates affect the performance considerably. Finally, to identify the gap to an optimal policy, the proposed approach is benchmarked against an optimal controller with perfect information. • Optimizing FACTS setpoints can substantially improve power flow and voltage control. • In practice, fixed setpoints are often used partly due to model-based challenges. • Accumulation of data motivates the use of model-free methods leveraging datasets. • We demonstrate performance gains with reinforcement learning that leverages datasets. • We show that dataset coverage and the number of pre-training updates impact results.