Q-Learning Inspired Geometric Shaping Optimization of APSK Constellations for the Nonlinear LEO-LEO Satellite Channels

The low-earth-orbit (LEO) satellite communication has been proposed to support the next generation 6G space-airground integrated network owing to its large coverage area and seamless connectivity. To overcome the challenge of long-distance transmission for the free space optical (FSO) links, satellite communication systems often rely on high power transmissions to achieve high signal-to-noise ratios (SNRs) and high throughput, albeit at the expense of increased nonlinearity. In this paper, we propose a Q-learning inspired geometric shaping (QLI-GS) optimization scheme for amplitude phase shift keying (APSK) constellation to mitigate nonlinear effects and maximize the transmission distance. A pairwise optimization (PO) algorithm inspired by Q-learning is employed to achieve QLI-GS optimization for the APSK constellation according to an enhanced nonlinear interference noise model (ENINM) with effective memory mechanism, which ensures fast convergence and suppresses nonlinear impairments. Our results show the iteration time of the QLI-GS-APSK is reduced by approximately 99.89% compared to the generalized mutual information (GMI) maximized GS algorithm, and it also offers greater nonlinear tolerance than normally multi-ring APSK and traditional POAPSK. By leveraging a fiber-optical sampled bidirectional dualterminal FSO equivalent link, the QLI-GS-32APSK can experimentally achieve a data rate of 100 Gbit/s over a 230 km distance based on a single-wavelength system. A maximum distance of 580 km at a data rate of 60 Gbit/s using QLI-GS8APSK has been successfully demonstrated under 7% hard decision-forward error correction (HD-FEC) threshold of 3.8e-3, with a corresponding link budget of 62 dB. Additionally, the robustness of QLI-GS optimization under different atmospheric turbulence conditions has been further validated. We demonstrate single carrier 100Gbit/s QLI-GS-32APSK rate transmission over a 4.6 km outdoor FSO link with an Fried's coherence length of 11 cm, achieving 0.9 dB shaping gain over 32APSK. The results highlight the promising potential of the proposed QLI-GS-APSK scheme in addressing nonlinear challenges in satellite communications.