Factorized Fourier Operator Based Neural Network for Fiber Nonlinearity Compensation in Coherent Optical Communication Systems

In long-haul and high-capacity coherent optical communications systems, it is essential to reduce the complexity of the underlying algorithms in the practical implementation of digital signal processing techniques for compensation of transmission impairments. Here, we investigate a factorized Fourier operator (F-FNO) based neural network (NN) for compensation the linear and nonlinear signal distortion in fiber simultaneously. The combination of the factorized Fourier transforms and residual connection enables the F-FNO based NN to compensate fiber impairments with higher accuracy, lower computational complexity and faster convergence rate, compared with our previous Fourier operator (FNO) based NN. The time consumption is further decreased by sharing the Fourier-domain weight matrices. The feasibility of the proposed scheme is verified experimentally in 32GBaud polarization division multiplexed (PDM) uniformly distributed 16/32/64-ary quadrature amplitude modulation (PDM-16/32/64QAM) as well as probabilistically shaped 16/32/64QAM (PDM-PS-16/32/64QAM) coherent optical communication systems. The experimental results demonstrate that the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i>-factor improvement is extremely close to the standard digital back propagation (DBP) with 16 steps per span (16StPS-DBP) even in the highly nonlinear region, while with much lower computational complexity and time consumption.