Neural network-based nonlinear equalizer using sight focal optimization for PCS 128 QAM 1 Tbit/s optical transmission

We propose what we believe is a novel sight focal loss (SFL) and an SFL optimization-based neural network nonlinear equalizer (SFL-NNLE), which extends conventional machine learning methods in high-speed coherent optical systems to improve the training efficiency of NNLEs. The SFL optimization effectively leverages the spatial information and emphasizes a stronger focus on the boundary-proximal hard samples to dynamically prioritize boundary-sensitive misclassification, thereby mitigating the overfitting and vanishing gradient induced by conventional loss functions. The effectiveness of the proposed SFL-NNLE to nonlinear impairments induced from optical/electrical components and fiber is experimentally demonstrated in our dual-polarization (DP)-16-ary quadrature amplitude modulation (QAM) back-to-back (BTB) and 150 km standard single-mode fiber (SSMF) transmissions. It achieves an optical signal-to-noise ratio (OSNR) gain of 1.45 dB at the 24% soft-decision forward error correction threshold of 4.5e-2 in a 118-GBaud DP-probabilistic constellation shaping (PCS) 128 QAM BTB transmission, realizing a net bit rate exceeding 1 Tbps.