Intelligent multi-agent Q-learning-based energy management of the grid-tied multi-carrier microgrid

The scientific aim of this paper is developing day-ahead energy management of grid-tied multi-carrier microgrid (MG) using the Q-learning method. Q-learning is employed to enable the agents to use their accumulated experience to determine the optimal price bids and power quantities in order to maximize their profit. The action space for power producer agents comprises the quantity of power along with associated price bid. The action for customer agents is defined by their power consumption level. Customer agents are categorized into tree types: 1) electrical, 2) thermal, and 3) integrated electrical-thermal. Following the clearance of the integrated heat and electricity auction market, power producer agents sell their surplus electrical power to the upstream grid. The proposed method is evaluated under three operation scenarios (OS) with two operating modes for critical loads: unconstrained and constrained. The results from OS2 show that customer agents learn to reduce their costs by 4.1 % (unconstrained) and 5.5 % (constrained), respectively, compared to OS1. Although, the income of the integrated energy storage (ES) with wind turbine agent increases by 9 % and 7 %, in the two operating modes, respectively. Likewise, the integrated ES with photovoltaic resource income increases by 1 % and 1.7 %. In the third scenario, which is designed to improve MG resilience, total customers cost increases by 5.7 % compared to OS2. Consequently, the revenues of Diesel, microturbine, and fuel cell agents increase by 14 %, 1.1 %, and 19 %, respectively. Numerical results demonstrate the effectiveness and applicability of the proposed method under various scenarios and operating conditions.