Joint Beamforming Design in Multi-STAR-RISs-Aided Cell-Free Massive MIMO Networks

Multiple simultaneously transmitting and reflecting reconfigurable intelligent surfaces (STAR-RISs) aided cell-free massive multiple-input multiple-output (CF mMIMO) network is investigated. A long-term sum-rate maximization problem is formulated to jointly optimize the active beamforming at each access point (AP) and the passive beamforming at each STAR-RIS while satisfying the quality of service (QoS) requirements. To address this non-convex problem with challenges caused by user mobility and high-dimensional optimization variables, two deep reinforcement learning (DRL)-based beamforming algorithms are proposed. Firstly, a soft actor-critic (SAC)-based centralized joint beamforming algorithm is proposed, which adds maximum entropy term to the objective function and provides an exceptional exploration-exploitation trade-off. However, since the centralized scheme might suffer from high communication loads and latency, we generalize the proposed approach into a distributed control. A multi-agent SAC (MASAC)-based distributed beamforming algorithm is further proposed, which employs the centralized training and decentralized execution (CTDE) framework, where each AP plays a role of an agent and makes decisions based on local observations. Simulation results demonstrate that: 1) The multiple STAR-RISs can effectively improve the performance of the CF mMIMO networks compared to other baseline schemes; 2) Both SAC-based and MASAC-based beamforming algorithms can maximize the sum-rate and guarantee the QoS of users in the long term; and 3) The MASAC-based beamforming algorithm performs better and converges faster than the SAC-based beamforming algorithm as it reduces the overall complexity and alleviates pressure on the fronthaul link transmission by making decentralized decisions.