Low complexity sphere decoding with probabilistic radius prediction and hybrid modulation for low latency wireless MIMO systems

Sphere Decoding is an important technique for detecting signals in MIMO systems. It is renowned for its near-optimal performance but is restricted by high computational complexity. This work proposes an improved framework, integrating Gaussian Approximation based radius prediction with hybrid modulation to overcome this limitation. To balance performance and complexity, the adaptive Gaussian model dynamically estimates the decoding radius based on channel conditions. This reduces the number of visited nodes and enhances computational efficiency. Building on this, a hybrid modulation scheme integrating BPSK and 16-QAM is used to further enhance error performance by adjusting modulation levels to varying channel qualities. The proposed method is simulated on an 8$$\times$$8 MIMO system using 16-QAM and hybrid modulation using BPSK and 16-QAM modulations. This Hybrid modulated MIMO system with Gaussian Approximation technique attains a drastic enhancement in system performance, reducing BER by up to 99.5%, and node complexity by 77.6% at 5 dB SNR compared to conventional Chi-Square approach. These results highlight the effectiveness of the two-tiered approach as a feasible, low-complexity, and low-latency solution for efficient detection, with promising applications in low-SNR and disaster-affected scenarios. Computationally efficient and low-energy algorithms are also crucial in 6G and IoT applications including massive machine-type communications and ultra-reliable low-latency communications which conserves battery power while supporting real-time functionality.