A Hybrid Deep Learning-Based Modeling Methods for Atmosphere Turbulence in Free Space Optical Communications

Free-space optical (FSO) communication provides high-capacity and secure links but is strongly impaired by atmospheric turbulence, which induces multi-scale irradiance fluctuations. Traditional approaches such as adaptive optics, multi-aperture and multiple-input multiple-output FSO schemes offer limited robustness under rapidly varying turbulence, while statistical fading models such as log-normal and Gamma–Gamma cannot represent multi-scale temporal correlations. This work proposes a hybrid deep learning framework that explicitly separates high-frequency scintillation and low-frequency power drift through a conditional variational autoencoder and a bidirectional long short-term memory dual-branch architecture with an adaptive gating mechanism. Trained on OptiSystem-generated datasets, the model accurately reconstructs irradiance distributions and temporal dynamics. For model-assisted signal compensation, it achieves an average 79% bit-error-rate (BER) reduction across all simulated scenarios compared with conventional thresholding and Gamma–Gamma maximum a posteriori detection. Transfer learning further enables efficient adaptation to new turbulence conditions with minimal retraining. Experimental validation shows that the compensated BER approaches near-zero, yielding significant improvement over traditional detection. These results demonstrate an effective and adaptive solution for turbulence-impaired FSO links.