Photonic THz Communication System Based on Geometric Shaping via End-to-End Autoencoder Learning and Pruning Algorithm

We propose a novel nonlinearity compensation method that integrates an end-to-end autoencoder (AE) model with structured pruning (SP) technology for geometric shaping (GS) in a 64QAM 400GHz photonic THz communication system. The results show that this approach not only enhances the Q-factor by 1.25 dB but also extends the transmission distance by 30 km compared to unshaped constellation schemes at optimal launch power. Additionally, this method significantly reduces computational complexity compared to the standard AE-GS scheme.