End-to-end optimization based on residual neural networks for improved optical fiber communication

High-capacity optical fiber transmissions increasingly face performance limits imposed by nonlinear transmission effects. This paper examines deep learning-based enhancements in optical fiber communication through an end-to-end autoencoder (AE) framework. A residual neural network (ResNet) architecture is employed to replace traditional dense layers, which uses shortcut connections for improved learning under nonlinear channel impairments. Using the achievable information rate (AIR) as the performance metric, the ResNet-based AE demonstrates 2.6% superior compared to dense-layer AE under high nonlinearity conditions. Although dense layer AEs offer lower complexity, the ResNet achieves 3.75% Q-factor improvements, making it a compelling choice for next-generation high-speed optical systems.