Enhancing reliability and spectral efficiency in future wireless networks via intelligent omni-surface enhanced MU-MIMO cooperative hybrid NOMA

The growing demand for high-capacity next-generation networks drives the need for efficient and scalable multiple access schemes. This study proposes a unified integrated analytical framework for intelligent omni-surface (IOS)-enhanced multi-user MIMO cooperative hybrid NOMA (MU-MIMO-C-HNOMA) system employing maximal ratio transmission (MRT) at the gNodeB (gNB) to enhance reliability, throughput, and sum spectral efficiency (SSE). Closed-form mathematical models for outage probability and throughput expressions are derived, and an optimal power allocation strategy is formulated to maximize SSE under the dual impairments due to imperfect successive interference cancellation (iSIC) and hardware distortions (HWD). Crucially, the analysis establishes a compelling energy-efficient scaling paradigm, revealing that leveraging passive IOS elements is fundamentally superior to scaling active transmit antennas. Furthermore, the study validates real-world feasibility by demonstrating that practical discrete-phase and blind-IOS configurations offer robust performance that asymptotically approaches the ideal benchmarks. The proposed system with strong-weak strong-weak (SWSW) user pairing achieves SNR gains of approximately 0.36–3.14 dB for weaker user equipment (WUE) and 0.13–2.5 dB for stronger UE (SUE) at a target outage of $$10^{-3}$$ resulting in throughput gains of about 0.03 - 0.33 bps/Hz and SSE gains of 0.02 - 0.14 bps/Hz over other pairing schemes. Furthermore, optimal power allocation achieves 0.48 - 1.02 bps/Hz higher SSE than sub-optimal and minimum power allocation schemes, confirming robustness against dual-impairments due to iSIC and HWD effects. Compared to the conventional IOS-aided HNOMA system, the proposed framework demonstrates superior scalability, reliability, and performance, achieving up to 0.31 bps/Hz SSE improvement for two antennas, confirming its potential as a highly promising solution for next-generation wireless networks.