Joint Optimization of Geometric and Probabilistic Constellation Shaping for OFDM-ISAC Systems

6G communications systems are expected to integrate radar-like sensing capabilities enabling novel use cases. However, integrated sensing and communications (ISAC) introduces a trade-off between communications and sensing performance because the optimal constellations for each task differ. In this paper, we compare geometric, probabilistic and joint constellation shaping for orthogonal frequency division multiplexing (OFDM)-ISAC systems using an autoencoder (AE) framework. We first derive the constellation-dependent detection probability and propose a novel loss function to include the sensing performance in the AE framework. Our simulation results demonstrate that constellation shaping enables a dynamic trade-off between communications and sensing. Depending on whether sensing or communications performance is prioritized, geometric or probabilistic constellation shaping is preferred. Joint constellation shaping combines the advantages of geometric and probabilistic shaping, significantly outperforming legacy modulation formats.