Attention-enhanced graph neural networks for nonlinearity compensation in optical fiber communication

Abstract Nonlinear impairments remain the dominant limitations and key challenges for researcher in high-capacity communication system especially optical fiber communication. The traditional digital signal processing (DSP) approached is limited due to its working structure on linear approximation of the Kerr effect. As optical network is also utilized to support 5G/6G backhaul, cloud computing infrastructure, and ecosystems, the demand for intelligent and adaptive compensation scheme has demand its strength significantly. This paper presents AEGNN (attention-enhanced graph neural network), framework that integrated Graph Attention Networks (GAT) with multi-head self-attention mechanisms to model space-time topology of optical networks. The proposed article addresses nonlinearity compensations along with optical performance monitoring (OPM) and dynamic resource allocation in elastic optical networks (EONs) within a unified multi-task learning formulation. Three opensource datasets ensure full reproducibility: OptiCommPy (synthetic fiber propagation), OpticalNet (experimental signal impairments), and the ITU-T AI/ML benchmark (BER/ Q -factor standardized benchmarks). All experiments were repeated five times with different random seeds; results are reported as mean ± standard deviation. AEGNN achieves R 2 = 0.963 ± 0.005 for Q -factor prediction and MSE = 0.0089 ± 0.0004 on the OptiCommPy dataset, reducing MSE by 78.4 % relative to DSP, 61.5 % relative to CNN, and 55.1 % relative to Transformer baselines. Additional metrics RMSE = 0.341 dB and MAE = 0.228 dB confirm practical suitability for real-time OPM (industry requirement: RMSE < 0.5 dB). To the best of effort and knowledge, this is the first fully reproducible framework combining attention-based GNNs with open-source optical communication datasets. The trained model weights and preprocessed datasets are to facilitate future adoption.