Hybrid Beamforming and Deep-Learning-Enabled Precoding for O-RAN mmWave Massive MIMO

This work investigates cellular millimeter-wave (mmWave) massive multiple-input multiple-output (MIMO) systems within the open radio access network (O-RAN) architecture, integrating the compatible spectrum, air interface, and networking entities of beyond fifth-generation wireless networks. To overcome O-RAN fronthaul (O-FH) load limitations and the short wavelength inherent in mmWave bands, we design a hybrid beamforming architecture with digital and analog beamformers generated at the O-RAN distributed unit and O-RAN radio unit, respectively. Using the information theory, we develop non-grid-of-beams analog beamformers to maximize the sum-spectral efficiency (SE) under constant-modulus constraints. For digital precoding, we apply a successive convex approximation method with second-order cone program procedures to maximize sum-SE, while addressing transmit power and limited O-FH load constraints, and ensuring user quality of service requirements. Sub-optimal digital combiners are also designed based on the inherent characteristics of the user side. However, the current optimization approach suffers from long execution times, posing challenges for near-real-time beamforming configurations. To address this issue, we propose an efficient deep learning (DL)-based digital precoding scheme with short execution time, low computational complexity, and high performance. Numerical results demonstrate that the proposed DL-based precoding scheme provides superior performance compared to benchmark schemes, generalizes well to environments with imperfect CSI and user mobility, and scales effectively to massive MIMO configurations.