Reinforcement Learning-Based Low-Level Control Strategy for Modular Multilevel Converters

This paper presents a novel reinforcement learning (RL)-based low-level control strategy for modular multilevel converters (MMCs), addressing critical operational challenges such as real-time sub-module (SM) energy stabilization, voltage tracking, and switching loss minimization. Traditional control strategies, including low-frequency modulation methods like Nearest Level Modulation (NLM) and PWM-based techniques such as Phase Shifted Carrier (PSC), face limitations in harmonic suppression, computational complexity, and efficiency in highvoltage scenarios. Leveraging the data-driven capabilities of RL, this study introduces a Deep Q-Network (DQN)-driven control framework that autonomously optimizes SM switching sequences without requiring precise mathematical models. The proposed strategy integrates an 11-dimensional observation space, encompassing stack voltage references, real-time currents, and capacitor voltages, and employs a reward function that balances voltage tracking precision, capacitor voltage balancing, and switching loss reduction. Simulation experiments on a single-phase MMC demonstrate the effectiveness of the approach in enhancing system performance and reliability. This research pioneers the application of RL in MMC control, offering a scalable and efficient solution for modern power systems.