Performance Analysis of a Coherent Heterodyne Detection Hybrid FSO/RF System Based on Nakagami-m Fading Channel and $\mathcal{F}$ Atmospheric Turbulence Channel

In this paper, we investigate the performance of a hybrid Free Space Optical/Radio Frequency (FSO/RF) communication system using coherent heterodyne detection and selection combining technique. The FSO link employs the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathcal{F}$</tex> channel model, considering pointing errors; while the RF link uses the Nakagami-m fading channel model. We derive expressions for the average bit error rate (BER) and outage probability of the hybrid FSO/RF system employing subcarrier intensity modulation (SIM) and coherent heterodyne detection (HD). We obtain closed-form analytical expressions using the Meijer G function and the extended generalized bivariate Meijer G function. We analyze the average BER and outage probability for both hybrid FSO/RF systems and single FSO links under various subcarrier modulation schemes, turbulence intensities, pointing errors, and RF channel fading parameters. Simulation results demonstrate that the hybrid FSO/RF communication system can improve system performance to a certain extent.