Deep Q-Network Optimized Two-Stage Hybrid Beamforming for Ultra-Reliable Millimeter-Wave and Terahertz Links in 6G Systems

The rapid development of the 6G network makes the integration of millimeter wave (mmWave) and terahertz (THz) communication an important foundation for achieving ultra-high speed and ultra-low latency communication, but it also presents difficult challenges including difficult beam alignment, large propagation loss, and rapid channel variation, which hinders the reliability and stability of the system. To address the above issues, this paper proposes a new DQN-UR-HBF (ultra-reliable hybrid beamforming based on deep Q network) optimization framework, which extends and innovates the existing hybrid analog-digital beamforming structure, and incorporates a double-stage optimization mechanism of deep reinforcement learning. The first stage uses the environment perception signal index to select the adaptive analog beamcodebook, and the second stage uses a multidimensional reward function to optimize the digital precoding matrix, and the reward function considers the beam gain, interference rejection and reliability requirements at the same time. In addition, the preferential experience playback mechanism, the double-Q network update strategy and the spatiotemporal attention state encoder are introduced to accurately capture the dynamic change characteristics of millimeter-wave and terahertz channels in the high-speed movement and occlusion environment of users. The experimental results demonstrate that compared to traditional DQN-HBF model, the proposed DQN-UR-HBF model achieves the success rate of beam alignment in high-speed scenario and strong occlusion which is improved by 18.6%.