Joint Rate Splitting and Secure Beamforming Optimization for RIS-RSMA-Enabled Integrated Sensing and Communication

This paper proposes a novel reconfigurable intelligent surface (RIS) and rate-splitting multiple-access (RSMA)-assisted integrated sensing and communication (ISAC) system (i.e., the RIS-RSMA-ISAC scheme) to strengthen the physical-layer security (PLS) while ensuring its required sensing performance is maintained. Specifically, RSMA is served as a flexible means of managing multi-user interference via common and private message transmission, while RIS is exploited to adaptively shape the propagation conditions to support secure transmission and target sensing. A secrecy-sum-rate maximization problem is formulated for legitimate users through the joint optimization of transmit beamforming at the base station (BS), reflection coefficients at the RIS, and secrecy rate allocation, while considering the transmit power limit at the BS, RIS reflection constraints, quality-of-service (QoS) requirements of legitimate users, along with a minimum sensing beam-pattern gain toward the target direction. An efficient algorithm is developed to tackle the highly coupled non-convex problem, which relies on successive convex approximation (SCA) and alternating optimization (AO). With an identical transmit power budget, numerical results demonstrate that, compared to the RIS-NOMA-ISAC and RIS-RSMA-Random schemes, the proposed RIS-RSMA-assisted ISAC scheme achieves approximately 6% and 35% improvements in secrecy sum rate, respectively. The performance gains become more evident under stringent sensing constraints and different system configurations, validating the efficacy of our joint optimization framework.