Reinforcement Learning for Optimized Multi-Use Operation of Battery Energy Storage Systems

The increasing integration of renewable energy into power systems results in increased volatility, which demands greater flexibility in future system designs to ensure grid security and reduce market prices. This paper introduces a Reinforcement Learning (RL) agent for a Battery Energy Storage System (BESS) that accounts for battery-specific degradation and efficiency while operating across real-time markets for multiple applications. The BESS, as an extremely fast and flexible storage system, can serve as a versatile 'buffer', modulating generation and consumption. Furthermore, it has the ability to participate in several markets simultaneously, also referred to as Multi-Use Operation. The BESS experiences degradation during its operation, a complex process characterized by non-linear dependencies that lack a straightforward mathematical representation. Although there are various methodologies to model battery-specific properties, it is essential to implement a model capable of optimizing real-time trading across different markets while taking degradation into account. We have developed an RL model with a non-uniform structure that allows for trading at various times within the same period, considering degradation effects. Our simulations involved varying and analyzing battery-specific properties, with results compared to benchmarks from two different Mixed-Integer Linear Programming (MILP) approaches. Our BESS RL optimization successfully identifies profitable patterns for arbitrage trading in Intraday (ID) and Day-ahead (DA) markets, achieving 68% of the revenue potential relative to the theoretical optimum. Furthermore, when factoring in degradation costs, our BESS model yielded higher profits than benchmark modeling for an active market participant.