Intelligent Omni-Surface-Assisted Cooperative Hybrid Non-Orthogonal Multiple Access: Enhancing Spectral Efficiency Under Imperfect Successive Interference Cancellation and Hardware Distortions

Non-orthogonal multiple access (NOMA) has emerged as a key enabler of massive connectivity in next-generation wireless networks. However, conventional NOMA studies predominantly focus on two-user scenarios, limiting their scalability in practical multi-user environments. A critical challenge in these systems is error propagation in successive interference cancellation (SIC), which is further exacerbated by hardware distortions (HWDs). Hybrid NOMA (HNOMA) mitigates SIC errors and reduces system complexity, yet cell-edge users (CEUs) continue to experience degraded sum spectral efficiency (SSE) and throughput. Cooperative NOMA (C-NOMA) enhances CEU performance through retransmissions but incurs higher energy consumption. To address these limitations, this study integrates intelligent omni-surfaces (IOSs) into a cooperative hybrid NOMA (C-HNOMA) framework to enhance retransmission efficiency and extend network coverage. The closed-form expressions for average outage probability and throughput are derived, and a power allocation (PA) optimization framework is proposed to maximize SSE, with validation through Monte Carlo simulations. The introduction of a novel strong-weak strong-weak (SW-SW) user pairing strategy capitalizes on channel diversity, achieving an SSE improvement of ∼0.48% to ∼3.81% over conventional pairing schemes. Moreover, the proposed system demonstrates significant performance gains as the number of IOS elements increases, even under imperfect SIC (iSIC) and HWD conditions. By optimizing PA values, SSE is further enhanced by at least 2.24%, even with an SIC error of 0.01 and an HWD level of 8%. These results underscore the potential of an IOS-assisted C-HNOMA system with SW-SW pairing as a viable solution for improving multi-user connectivity, SSE, and system robustness in future wireless communication networks.