Joint Transmitter-Receiver Optimization for Optical Communication over Nonlinear Channels

In linear additive white Gaussian noise (AWGN) channels, optimal signaling schemes can be derived directly from established theoretical models. However, fiber-optic channels are nonlinear, making it challenging to derive optimal signaling schemes analytically. Additionally, the nonlinear behavior of electro-optic modulators and lasers under direct modulation further complicate signal optimization in fiber-optic communication. This paper demonstrates how machine learning techniques can be leveraged to jointly optimize constellations, pulse shaping, and receiver filters for fiber-optic channels. By learning optimal signal strategies tailored to specific channel characteristics, significant performance improvements are achievable.