Multi-Objective Optimization of Machine Learning-Based Nonlinear Equalizers for Digital Coherent Optical Interconnects

The growing demand for high-capacity optical networks has driven the advancement of digital coherent optical communication systems, which rely on sophisticated signal processing to mitigate transmission impairments, including nonlinear fiber distortions. Traditional nonlinear compensation techniques, such as the Inverse Volterra Series Transfer Function (IVSTF) and Digital Back Propagation (DBP), are computationally expensive and require oversampling. Machine learning-based equalizers offer reduced complexity and improved adaptability. However, for dispersion-uncompensated systems with a high bandwidth-reach product, the computational complexity increases, becoming a significant concern. This work presents a multi-objective optimization approach for nonlinear equalization, balancing performance and computational cost. A sequentialized multilayer perceptron (MLP)-based nonlinear equalizer is applied to a 16-QAM digital coherent optical system over a 150 km uncompensated link at 112 Gbps. Various hyperparameter configurations, tap and neuron counts, are evaluated to identify optimal trade-offs. A Pareto front analysis quantifies the complexity-performance trade-off, providing insights into selecting an optimal equalizer configuration based on system constraints.