Adaptive AI in Smart Grid: A Continual Reinforcement Learning Framework for Cyber-Physical Systems

The shift toward smart cities emphasizes sustainability, resilience, and efficient resource use, with smart grids at the forefront, integrating real-time communication to manage peak demand and support renewable energy integration. Central to this transformation are cyber-physical systems (CPS) that enable realtime monitoring and adaptive control within the grid. In CPS architecture, Digital Twin (DT) plays a vital role by creating high-fidelity virtual replicas of physical grid components, enabling real-time simulation and prediction of system behaviors. This paper introduces a novel adaptive AI framework for continual reinforcement learning (CRL) to enhance CPS-driven smart grid operations. CRL is crucial in this context as it allows for ongoing adaptation to dynamic environments, addressing the limitations of traditional reinforcement learning that typically relies on static training paradigms. The framework is built on the Deep Deterministic Policy Gradient (DDPG) algorithm, chosen for its effectiveness in handling continuous action spaces and complex energy management decisions. By incorporating a Parameter Adaptation Module as the adaptive AI controller-named PAM-CDDPG-the framework dynamically adjusts reinforcement learning parameters to respond to fluctuating energy dynamics. The framework's effectiveness is evaluated through comprehensive smart grid DT testbed named Citylearn, which provides accurate virtual representations of urban energy systems. This DT based evaluation framework assesses the system's capacity to reduce total positive net electricity consumption (NEC) across diverse scenarios. Results indicate substantial improvements in energy efficiency and demonstrate the proposed framework's transferability between zones, supported by PAM's adaptability to distinct energy profiles and validated through the high-fidelity Digital Twin simulation environment.