Comparative Analysis of Optimal Control and Reinforcement Learning Methods for Energy Storage Management Under Uncertainty

The challenge of optimally controlling energy storage systems under uncertainty conditions, whether due to uncertain storage device dynamics or load signal variability, is well established. Recent research works tackle this problem using two primary approaches: optimal control methods, such as stochastic dynamic programming, and data-driven techniques. This work’s objective is to quantify the inherent trade-offs between these methodologies and identify their respective strengths and weaknesses across different scenarios. We evaluate the degradation of performance, measured by increased operational costs, when a reinforcement learning policy is adopted instead of an optimal control policy, such as dynamic programming, Pontryagin’s minimum principle, or the Shortest-Path method. Our study examines three increasingly intricate use cases: ideal storage units, storage units with losses, and lossy storage units integrated with transmission line losses. For each scenario, we compare the performance of a representative optimal control technique against a reinforcement learning approach, seeking to establish broader comparative insights.