Hierarchical Reinforcement Learning for Degradation-Aware and Coordinated Control of Virtual Power Plants

Abstract Virtual Power Plants (VPPs) coordinate distributed energy resources (DERs) under forecast uncertainty and multi-time-scale constraints, while battery and generator wear can erase short-term trading gains. This paper proposes a hierarchical reinforcement learning (HRL) controller that matches aggregator practice: a slow-time-scale manager issues location-level power subgoals together with a continuous autonomy signal, and fast-time-scale workers dispatch devices under local physics. Battery and generator degradation enter the training reward through a three-phase curriculum that activates a throughput-budget penalty and a cycle-depth surcharge, so the policy optimizes profit net of wear rather than arbitrage alone. On a 100-day shared-seed benchmark against two decentralized RL controllers, an open-loop stochastic mixed-integer programming (MIP) day-ahead optimizer, and a closed-loop model predictive control (MPC) baseline that reoptimizes hourly over an 8-h horizon, HRL attains an average daily profit of \$50,264─3.1× MPC and 4.0× MIP─at 50–170 ms per simulated day. HRL is the only method whose throughput-to-cycle-depth ratio exceeds unity (1.18), and it incurs only \$24.6 of cycle-depth-aware wear per \$1,000 of profit (2.5× better than MPC). These results indicate that degradation-aware hierarchical control can outperform both open-loop stochastic optimization and receding-horizon MPC for real-time VPP dispatch.