Beamforming Design for Beyond Diagonal RIS-Aided Cell-Free Massive MIMO Systems

Reconfigurable intelligent surface (RIS)-aided cell-free (CF) massive multiple-input multiple-output (mMIMO) is a promising technology for further improving spectral efficiency (SE) with low cost and power consumption. However, conventional RIS has inevitable limitations due to its diagonal scattering matrix. In contrast, beyond-diagonal RIS (BD-RIS) has gained great attention. This correspondence focuses on integrating a hybrid transmitting and reflecting BD-RIS into CF mMIMO systems to enhance coverage and spatial multiplexing. This requires completing the beamforming design under the transmit power constraints and unitary constraints of the BD-RIS, by optimizing active and passive beamformer simultaneously. To tackle this issue, we introduce an alternating optimization algorithm that decomposes it using fractional programming and solves the subproblems alternatively. Moreover, to address the challenge introduced by the unitary constraint on the beamforming matrix of the BD-RIS, we propose a Riemannian limited-memory Broyden-Fletcher-Goldfarb-Shanno (R-L-BFGS) algorithm to solve the problem optimally. Simulation results show that our algorithm achieves faster convergence and finds higher-quality solutions compared with baselines, while also demonstrating a favorable performance-complexity trade-off.