Multi-Agent DRL for Distributed Codebook Design in RIS-Aided Cell-Free Massive MIMO Networks

This paper proposes an innovative approach for enhancing network capacity and coverage by integrating cell-free massive multiple-input multiple-output (CF-mMIMO) networks with reconfigurable intelligent surfaces (RISs). A significant challenge in leveraging RIS-assisted CF-mMIMO lies in the cooperative beam training across multiple access points (APs) and RISs, complicated by the passive nature of reflective elements and the complexity channel state information (CSI) acquisition in millimeter wave mMIMO systems. To address these challenges, we develop a multi-agent deep reinforcement learning (MA-DRL) framework that jointly designs beamforming and reflection codebooks for distributed APs and RISs, eliminating the need for CSI and relying solely on received power measurements feedback. The joint beamforming and reflection codebook design problem is decomposed into two sub-problems: one for beam codebook design at APs and another for sequential reflection codebook design at RISs. We employ transfer learning to speed up learning convergence and reduce computational complexity for training multiple RISs. Additionally, we introduce an AP and RIS selection scheme that improves overall energy efficiency and reduces backhaul overhead. Extensive simulations demonstrate that our proposed MA-DRL approach curtails number of beams significantly, thereby outperforming the widely adopted discrete Fourier transform (DFT) codebooks by achieving an 84% reduction in beam training overhead. Our findings suggest that increasing the number of passive RISs allows putting more APs into idle mode, leading to substantial savings in hardware and energy costs.