Deep Reinforcement Learning-based Stackelberg Game for Energy Trading in Power Grid Metaverses

Metaverses are a virtual space that has revolutionized many fields, such as healthcare and education. By integrating metaverses with power grids, power grid metaverses are regarded as a digital representation of the power grid infrastructure, giving rise to some potential applications, such as visualization and monitoring of energy trading. Energy balancing is essential to ensure that the supply of electricity matches the demand. Power grid metaverses can be used to identify potential imbalances between supply and demand and execute energy trading in real time, ensuring the proper functioning of smart grids. However, without reliable incentives, energy trading may not be conducted effectively. Considering the energy trading between EVCOs (EVCOs) and EMs (EMs), we propose a Stackelberg game model where the EM acts as the leader, setting the electric energy price, while EVCOs follow. We utilize a soft actor-critic algorithm to identify the Stackelberg equilibrium, addressing non-stationary challenges through the design of a partially observable policy, definition of the action space, and formulation of the reward function. Numerical results demonstrate that the proposed algorithm effectively achieves the Stackelberg equilibrium, indicating the robustness of our approach.