Hybrid Beamforming with Joint Deep Reinforcement Learning and Unfolding Networks for Integrated Sensing and Communication Systems

The integrated sensing and communication (ISAC) technology has gained increasing attention in recent years due to its excellent performance of increasing the spectrum and hardware efficiencies. In this paper, we investigate the joint optimization of beam selection and digital beamforming for a millimeter-wave (mmWave) ISAC system to simultaneously improve the performance of communication and sensing. We propose a novel hybrid beamforming scheme based on deep learning by maximizing the sum of communication mutual information (CMI) and sensing mutual information (SMI) to enable multi-user multiple-input multiple-output (MU-MIMO) communication and multiple-input single-output (MISO) radar sensing. Specially, we propose a joint deep reinforcement learning and unfolding network (DRL-UN) to optimize the beam selection and digital beamforming matrices at the base station (BS) in an ISAC system. Simulation results demonstrate that the proposed hybrid beamforming scheme significantly outperforms the existing algorithms in terms of sensing and communication (S&C) sum-rate in a mmWave ISAC system.