Optimizing Codebook Design in mmWave Massive MIMO Using Digital Twin-Assisted DRL

The inefficiency of predefined codebooks poses significant challenges for millimeter-wave (mmWave) massive multiple-input multiple-output (MIMO) systems in dynamic environments, characterized by excessive beam training overhead and suboptimal beamforming performance. While deep reinforcement learning (DRL) has emerged as a promising alternative for adaptive codebook design, its reliance on extensive real-world interactions hinders practical deployment. To overcome this limitation, we propose a digital twin-assisted DRL (DT-DRL) framework that integrates model-driven with data-driven learning. By embedding expert knowledge on signal composition features into the digital twin (DT) module, our framework implicitly reconstructs wireless environments using limited beamforming vectors and corresponding received power measurement, thereby eliminating the requirement for continuous interaction between DRL and the real-world. The DRL agent subsequently explores optimal beam patterns efficiently within this DT constructed virtual environment. Empirical results demonstrate that the DT-DRL generated codebooks achieve higher beamforming gains across diverse propagation scenarios compared to classical codebook. Furthermore, the proposed solutions significantly reduces resource consumption and pilot overhead relative to traditional DRL solutions.