Adaptive Autoencoder-Based Probabilistic Shaping in Nonlinear Optical Communication Systems Considering Equalization-Enhanced Phase Noise

This paper proposes an autoencoder (AE)-based probabilistic shaping (PS) framework for coherent optical fiber systems that, for the first time, explicitly incorporates equalization-enhanced phase noise (EEPN). By modeling EEPN with both a simplified analytical channel and a more accurate physical mechanism, we train AE-based PS distributions to maximize generalized mutual information (GMI). Our results reveal that the computationally efficient analytical model yields shaping gains nearly on par with its physically derived counterpart. Furthermore, PS distributions learned under linear assumptions retain robust performance in nonlinear fiber channels, particularly at higher laser linewidths, underscoring strong resilience to model mismatch. We also examine the adaptability of these AE-trained constellations across various system parameters, including linewidth and launch power, providing insights into their practical deployment. Overall, this work highlights the feasibility of EEPN-aware PS and demonstrates the effectiveness of simplified channel models for reducing design complexity, offering guidelines for implementing AE-based PS in nextgeneration coherent optical communication systems.