Multi-Agent Reinforcement Learning Based Distributed Multi-User Scheduling and Beamforming Design in Multi-Cell Systems

In multi-cell systems, joint multi-user scheduling and beamforming (MUS-BF) design are two fundamental problems that are usually studied separately in the existing literature. In this work, we focus on the joint MUS-BF design with the goal of minimizing the network experience delay. Since the objective function is long-term and non-convex, we first equate the network experience delay with the long-term average network experience delay cost, which can be optimized by reinforcement learning techniques. Then, we propose a distributed MUS-BF scheme based on multi-agent reinforcement learning to minimize the long-term average network experience delay cost, where the adaptive multi-deep Q network (DQN) architecture is designed at each base station (BS) to find suitable MUS-BF with low computational complexity. Each BS is able to train its own multi-DQN and implement appropriate MUS-BF based on the local information. Simulation results validate that the adaptive multi-DQN architecture decreases 70% computational complexity of the proposed algorithm without compromising performance compared to the non-adaptive design. Compared to existing schemes, the proposed scheme achieves superior performance with the lowest information overhead, making it a more appealing solution for real-time MUS-BF in delay-sensitive scenarios.