Performance-optimized hybrid multidomain dispersion compensation techniques for next-generation ultra-high capacity coherent optical communication systems

Abstract The coherent optical transmission system is expected to overcome linear and nonlinear transmission issues with the development of ultra-high capacity optical network to meet the needs of cloud computing, artificial intelligence, 5G/6G communication, and hyperscale data centers. Impregnable compensation for long-haul, high-speed transmissions is not possible with traditional methods of impairment compensation in the optical domain or in digital signal processing (DSP). The proposed performance-optimized hybrid multidomain compensation (MDC) scheme in this paper is based on optical domain dispersion compensation and advanced DSP algorithms to improve the transmission quality of coherent optical fiber communication systems. Architecture design was carried out in OptiSystem 23.0, and the implementation of the DSP algorithms was done in MATLAB R2024b. The following coherent 400 Gb/s transmission system was investigated, taking into account realistic transmission conditions, up to 800 km of standard single-mode fiber (SSMF). The simulation results show that the proposed hybrid compensation consistently provides the lowest BER and the highest Q-factor under various operating conditions and is close to the performance of the back-to-back reference system.