Enhanced approach for artificial neural network-based optical fiber channel modeling: Geometric constellation shaping WDM system as a case study

Recently, there has been increasing interest in applying machine learning (ML) approaches to enhance the performance of optical communication systems. This paper applies some of these approaches to design advanced wavelength-division multiplexed (WDM)-coherent optical fiber communication (OFC) systems assisted by the constellation shaping technique. A theoretical design and performance investigation are reported assuming end-to-end deep learning (E2EDL) autoencoder (AE)-assisted system configuration. A flexible artificial neural network (ANNs)-based optical fiber channel modeling approach suitable for different multi-span transmission links in OFCs is presented. This approach is applied to E2EDL-based geometric constellation shaping WDM systems and the results reveal that using a bi-directional gated recurrent unit (Bi-GRU)-neural network (NN) gives the best modeling that tracks the numerical nonlinear interference noise fiber model with much less computation time(~7%). This work is implemented using the Python programming language and utilizing the TensorFlow framework to develop the simulation models.