Enhanced OAM-FSO Links Through Optimized Multilayer Irregular RIS and Wavefront Correction Techniques

Twisted light, characterized by its Orbital Angular Momentum (OAM) and helical phase front, holds great promise for advancing Free-Space Optical (FSO) communication systems by significantly increasing both capacity and spectral efficiency. However, Atmospheric Turbulence induces power fluctuations and wavefront distortions, degrading system performance. To address these challenges, we propose a Multi-Layered Irregular-Reconfigurable Intelligent Surface (MLI-RIS) architecture designed to enhance data rates and mitigate the detrimental effects of turbulence. To optimize system performance while minimizing computational complexity, a two-stage optimization framework is introduced: (1) MLI-RIS topology design, and (2) joint phase-shift and precoding optimization. This framework leverages a Diversification Tabu–Advanced Neighborhood Search (DT-ANS) algorithm, along with a Neighborhood Exploration Strategy integrated with Simulated Annealing (NES-SA). Furthermore, a compressive sensing-based correction method using regular RIS is implemented to compensate for wavefront distortions and enhance the purity of OAM modes. Numerical simulations validate the effectiveness of the proposed approach in reducing error rates, mitigating wavefront distortions and crosstalk, and enhancing overall system performance. The results show that the proposed system achieves a Forward Error Correction level with a 1.4 dB loss compared to the ideal case, whereas the best of the other approaches considered in this paper achieve this level with an 18 dB loss compared to the ideal case.