Atmospheric turbulence-resilient hybrid MIMO RF/FSO communication systems: adaptive <i>N</i> -SM and OAM-OMI assisted <i>M</i> -ary SPPM modulation with advanced diversity multiplexing for next-generation wireless networks

Abstract This study presents a novel evaluation framework for optimizing free-space optical (FSO) communication systems by integrating multilevel modulation with orbital angular momentum (OAM)-based data stream multiplexing to maximize spectral efficiency (SE). The proposed architecture synergizes pulse-position modulation (PPM) and dense wavelength-division multiplexing (DWDM), while mitigating impairments from optical nonlinearities and atmospheric turbulence. Key innovations include the fusion of N -encoded adaptive spatial modulation (SM) with L -ary spatial PPM (SPPM) and a diversity strategy employing mixed multiple-input multiple-output (MIMO), multiple-input single-output in an M :1 ratio, and single-input single-output (SISO) configurations. Incorporating SM into FSO systems substantially reduces bit error rate by counteracting turbulence-induced signal degradation. The performance of SPPM, MIMO, and hybrid MIMO-SPPM/MPPM schemes is rigorously analyzed across RF, FSO, and hybrid RF/FSO channels under weak turbulence and dynamic atmospheric conditions. Simulations demonstrate that the hybrid MIMO/OAM-SPPM/MPPM model achieves an average symbol error probability of 10 −10 at 20 dB signal-to-noise ratio (SNR) – outperforming conventional approaches by orders of magnitude. Optimal SE is attained via hybrid modulation over hybrid RF/FSO links, with 4 × 4 MIMO-RF/DWDM-FSO configurations achieving 49 bits/s/Hz at 25 dB SNR and 55 bits/s/Hz at 17 dB SNR – surpassing SISO RF (16 bits/s/Hz) and FSO (21 bits/s/Hz) benchmarks.