Joint MCS Adaptation and Beamforming Design for Multiuser MISO Systems: A Constrained Hybrid Deep Reinforcement Learning Approach

This paper investigates the joint modulation-coding scheme (MCS) adaptation and beamforming design for multi-user multi-input single-output (MISO) systems, where one base station serves multiple user equipments (UEs) under imperfect and outdated channel state information (CSI). The sum-rate of the system is maximized while satisfying all UEs’ data rate requirements and the maximum transmit power constraint at the BS. Most existing beamforming designs overlooked that only a finite number of MCSs can be supported in practical communication systems. Moreover, previous works rely on perfect and real-time CSI for decision-making, neglecting processing delays and channel estimation errors. To circumvent the above issues, this paper puts forth an intelligent joint optimization scheme based on deep reinforcement learning (DRL) techniques. Specifically, a new DRL framework, termed constrained hybrid DRL (CHDRL), is first proposed, which incorporates Lagrangian primal-dual optimization theory and a constrained action selection policy into conventional DRL to tackle various constraints. By integrating deep Q-network (DQN) and deep deterministic policy gradient algorithms, CHDRL is capable of simultaneously optimizing MCS in the discrete action domain and beamforming in the continuous action domain. In addition, to handle the large discrete action space of DQN, we develop an action branch architecture for CHDRL to enable independent and concurrent MCS decisions at different UEs. Finally, a multi-parameter experience replay mechanism is designed to synchronously train Lagrangian multipliers and neural network parameters. Simulation results demonstrate that under imperfect and outdated CSI, CHDRL outperforms other benchmark schemes by (i) achieving a significantly higher sum-rate, (ii) meeting more UEs’ data rate requirements, and (iii) being more robust against different CSI delays and numbers of UEs.