Multi-Agent Reinforcement Learning for Sustainable Integration of Heterogeneous Resources in a Double-Sided Auction Market with Power Balance Incentive Mechanism

Traditional electricity market bidding typically focuses on unilateral structures, where independent energy storage units and flexible loads act merely as price takers. This reduces bidding motivation and weakens the balancing capability of regional power systems, thereby limiting the large-scale utilization of renewable energy. To address these challenges and support sustainable power system operation, this paper proposes a double-sided auction market strategy for heterogeneous multi-resource (HMR) participation based on multi-agent reinforcement learning (MARL). The framework explicitly considers the heterogeneous bidding and quantity reporting behaviors of renewable generation, flexible demand, and energy storage. An improved incentive mechanism is introduced to enhance real-time system power balance, thereby enabling higher renewable energy integration and reducing curtailment. To efficiently solve the market-clearing problem, an improved Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm is employed, along with a temporal-difference (TD) error-based prioritized experience replay mechanism to strengthen exploration. Case studies validate the effectiveness of the proposed approach in guiding heterogeneous resources toward cooperative bidding behaviors, improving market efficiency, and reinforcing the sustainable and resilient operation of future power systems.