Rotated Constellation-Based Probabilistic Shaping for Joint Radar Detection and Communication

Joint radar detection and communication enables environment-aware applications. Prior works focus on systems with infinite constellation waveforms. However, practical systems rely on discrete inputs, leading to performance degradation in both detection and communication due to the limited dimension of signal distribution. To address this, this paper focuses on compensating for the performance loss by adjusting both the geometry and probability of the discrete constellations. Specifically, considering the discrete inputs, the communication and detection performance are evaluated by mutual information and Kullback-Leibler divergence (KLD), respectively. Building upon the analysis, a rotated constellation-based probability shaping (RPS) scheme is proposed to design probability distribution and rotation angle by maximizing mutual information with power and KLD constraints. Given the non-convexity of the KLD constraint, a convex approximation-based block coordinate descent algorithm is proposed. Numerical results validate that the proposed RPS scheme effectively reduces performance loss.