A Deep Reinforcement Learning Framework for Joint Bs and Beyond Diagonal Ris Beamforming Design in Isac Systems

This paper investigates the joint optimization of the base station precoding matrix and the beyond-diagonal reconfigurable intelligent surface (BD-RIS) scattering matrix in an integrated sensing and communication (ISAC) system that employs rate-splitting multiple access (RSMA). To tackle the inherent nonconvexity and NP-hardness of the optimization problem and channel estimation uncertainties, we propose a novel deep reinforcement learning (DRL) framework, termed SAC-QA. The proposed SAC-QA algorithm leverages action quantization to discretize the continuous high-dimensional action space and incorporates a low-rank approximation with sparse correction for efficient BD-RIS matrix generation. Simulation results demonstrate that the proposed approach significantly improves the worst-case performance compared to conventional algorithms, thereby highlighting its potential for enabling robust and adaptive ISAC systems in next-generation wireless networks.