Accurate Closed-Form Approximation for Symbol Error Probability in Hexagonal QAM

Future communication systems are anticipated to facilitate applications requiring high data transmission rates while maintaining energy efficiency. Hexagonal quadrature amplitude modulation (HQAM) offers this owing to its compact symbol arrangement within the two-dimensional (2D) plane. Due to the structure of its hexagonal lattice, accurately calculating the error probability is highly challenging, if not entirely impractical. Building on the limitations of the current approaches, this paper presents a straightforward and precise approximation for calculating the symbol error probability (SEP) of HQAM in an additive white Gaussian noise channel. Simulations show the proposed approximation is highly accurate for all hexagonal QAM constellations. A novel cluster-based detection method with low computational complexity is proposed. The method ensures that its computational demand remains unaffected by increase in constellation order. Additionally, the average SEP performance of the system is evaluated under a practical scenario with imperfect channel state information over Nakagami-m fading channels.