Optimizing IRS placement and element configuration in B5G: A novel cooperative hybrid communication system

• The proposed system operates in two transmission phases. In the first phase, the source transmits signals to both the IRS and the relay; the IRS reflects the signal directly to the user. In the second phase, the relay decodes and forwards the signal to the user through both direct and IRS-assisted paths, ensuring robust multipath transmission. • Two novel optimization algorithms are developed to determine the optimal IRS placement and reflective element configuration under both fixed and flexible data rate constraints. These algorithms are essential for achieving high performance in terms of data rate and coverage across various deployment scenarios. • The proposed system is extensively evaluated under MISO-NOMA and MISO-OMA configurations. Numerical results demonstrate that the proposed cooperative hybrid system significantly outperforms traditional hybrid relay-IRS-aided and relay-dominant cooperative hybrid systems in terms of both data rate and EE. The performance is consistently superior across varying user distances and Tx power levels, validating the system’s robustness and scalability. The next generation of wireless networks demands transformative solutions to achieve ultra-reliable, high-capacity, and energy-efficient communications. Conventional systems based solely on relay-assisted transmission or intelligent reflecting surfaces (IRS) face inherent trade-offs in spectral efficiency, coverage, and energy consumption. In this work, we propose a novel cooperative hybrid communication system that integrates IRS with relay-assisted transmission to form a robust, scalable, and energy-aware design for beyond 5G (B5G) networks. Unlike traditional architectures, the proposed system enables mobile users to receive signals from the base station through three cooperative transmission paths: direct relay transmission, IRS reflection, and relay-assisted IRS reflection. This cooperative signal propagation enhances path diversity, improves link robustness, and increases spatial efficiency. Additionally, we introduce a joint optimization algorithm to determine the optimal IRS placement and reflecting element configuration, maximizing system throughput under practical rate constraints. Simulation results under Rayleigh fading conditions demonstrate the effectiveness of the proposed system across various deployment scenarios, including SISO, MISO-OMA, and MISO-NOMA. In the MISO-NOMA setting with 40 dBm transmit power, the system achieves a sum rate of 49.80 bits/s/Hz and an energy efficiency of 16.60 bits/Joule, outperforming benchmark hybrid relay-IRS-aided and relay-dominant cooperative hybrid systems. These findings establish the proposed system as a high-performing and energy-efficient solution for future wireless networks.