Optimized Hybrid Probabilistic and Geometric Constellation Shaping for Coherent Optical Communication Systems Using End‐to‐End Learning

To meet the growing demand for enhanced performance in coherent optical communication systems, increasing spectral efficiency and system capacity through constellation shaping is crucial. In this article, the end‐to‐end optimization of hybrid probabilistic and geometric constellation shaping (HPGS) under a Wiener phase noise channel is explored, enhanced by carrier phase estimation. By employing a differentiable two‐stage blind phase search algorithm integrated within digital signal processing (DSP) and utilizing gradient descent‐based back‐propagation, the approach ensures higher spectral efficiencies. Herein, the proposed method surpasses geometrically shaped 64QAM (QAM—quadrature amplitude modulation) by 0.082 bit per symbol in generalized mutual information at a 350 kHz linewidth. Additionally, the adaptivity of HPGS to higher‐order QAM formats, including 128QAM and 256QAM, is investigated, demonstrating significant performance gains. This research provides a cost‐effective solution for joint systematic optimization in optical communication systems, leveraging the differentiable channel model and receiver DSP.