A Novel, Accurate, Exact SEP Expression of HQAM Over AWGN Channel and Its Application Over Nakagami‐ <i>m</i> Fading Channel

ABSTRACT This paper proposes a simple and accurate exact expression for the symbol error probability (SEP) of general order hexagonal quadrature amplitude modulation (HQAM) in the presence of additive white Gaussian noise (AWGN) channel. The main concept is to derive a unique SEP expression using the concept of simplified decision regions. The proposed expression is accurate for very low as well as high signal‐to‐noise ratios, which is further validated with the help of Monte Carlo simulations. The significance of the proposed expression is further highlighted by simplifying the SEP expression of HQAM over Nakagami‐ fading channel, which includes the special case of Rayleigh fading (). To do so, we utilize an accurate approximation of the Gaussian function already available in the open literature.