Summary
This paper investigates the performance of a dual‐hop mixed relay system with radio frequency (RF) and free‐space optics (FSO) communication under the effect of pointing error (PE) and atmospheric turbulence (AT). This paper considers a system where RF and FSO links are cascaded. The RF link is modeled by Nakagami‐m fading, and the FSO link is modeled as gamma–gamma (G‐G) fading channel. Both the channel models use orthogonal frequency division multiplexing (OFDM) with M‐ary quadrature amplitude modulation (QAM). The expressions for probability density function, cumulative distribution function, signal‐to‐noise ratio, and ergodic capacity are derived. The moment generating function (MGF) of fading and the bit error rate (BER) of the OFDM‐based M‐ary QAM scheme is derived in terms of Meijer's G‐function. It has been observed that, in fixed gain relay systems, the modulation scheme's BER is dominated by the SNR of the RF link. While in a variable gain relay system, the turbulence conditions of the FSO system affect the SNR and the BER of the modulation method. The feasibility of heterodyne detection and intensity modulation direct detection (IM/DD) is analyzed in terms of outage probability and ergodic capacity. The results can be used to choose the optimal modulation order and relay system for QAM‐OFDM‐based optical wireless systems.