Performance evaluation of ANN-based dispersion compensation in optical communication systems

Abstract The increasing demand for high-capacity optical communication systems has highlighted the need for intelligent signal processing techniques capable of mitigating dispersion-induced impairments while maintaining low computational complexity. Conventional adaptive equalizers, such as the Least Mean Square (LMS) algorithm, exhibit limited performance under nonlinear channel conditions and varying signal distortions. This paper proposes an Artificial Neural Network (ANN)-based dispersion compensation approach for high-speed optical communication systems. The proposed model learns the nonlinear characteristics of the received signal and accurately reconstructs the transmitted data without requiring explicit channel modeling. System performance is evaluated using QPSK and 16QAM modulation formats through comprehensive analysis of bit error rate (BER), Q-factor, optical signal-to-noise ratio (OSNR), error vector magnitude (EVM), eye diagrams, constellation diagrams, received optical power, classification accuracy, regression analysis, and execution time. Simulation results demonstrate that the proposed ANN significantly outperforms the conventional LMS equalizer by reducing BER, improving Q-factor, decreasing EVM by more than 96 %, lowering the required OSNR at BER = 10 −3 , and accelerating execution time by approximately 2.44. These results confirm that AI-based equalization represents an efficient and reliable solution for next-generation intelligent optical communication systems.