Complex-valued deep neural network equalizer with an attention mechanism for the photonics-aided 400  GHz PS-16QAM system

Terahertz communication shows significant potential for 6G due to its ultra-wideband characteristics. Probabilistic shaping (PS) combined with quadrature amplitude modulation (QAM) improves the signal-to-noise ratio, enhances system capacity, and extends transmission distance. However, the use of PS modulation introduces class imbalance in QAM signals, which negatively affects the performance of traditional machine learning algorithms. To address this issue, we propose a nonlinear equalization method based on a complex-valued deep neural network, termed the complex-valued convolutional neural network-gated recurrent unit-attention (CV-CGA) model. This method accurately models the amplitude and phase characteristics of signals while effectively mitigating the class imbalance problem in PS signals. In a photonics-aided 400 GHz PS-16QAM THz communication system, CV-CGA significantly reduces the bit error rate and enhances system robustness, particularly in high-power and nonlinear environments. Compared to traditional neural network equalizers, CV-CGA offers improved accuracy in signal recovery, as well as better stability and generalization capabilities.