Optimal AC Power Flow with Energy Storage and Renewable Energy: An Efficient RL Algorithm Capable of Handling Equality Constraints

Using energy storage to solve the multiperiod OPF problem for renewable energy fluctuation is an effective way to increase operation safety and reduce the cost of power systems. However, in solving this OPF problem, model-based methods cannot accurately model uncertain scenarios, while traditional RL methods cannot satisfy the constraints well, and both methods have limitations. Therefore, we propose an RL method, ERL-HC, that does not require scene modelling and can handle general forms of physical constraints. First, a constraint policy network (CPN) is proposed that corrects the output of a neural network on the basis of the inequality generalized reduced gradient (GRG) method; the outputs of this network satisfy all constraints, and it can be trained in an end-to-end manner. Second, the critic network is improved based on the IM method to increase the sample learning efficiency by improving the agent's understanding of state interdependencies. Finally, the adaptive-tuning Lagrange multiplier method is applied in the AC framework to reduce the number of iterations of the inequality GRG in the CPN and efficiently train ERL-HC. ERL-HC was tested on two systems of different sizes. The results show that ERL-HC has a better learning ability than general safe RL algorithms, overcomes the limitations of mainstream safe RL methods in handling equality constraints, and addresses the poor generalization issues of RL methods that can handle equality constraints.