Enhancing the Uplink of Cell-Free Massive MIMO through Prioritized Sampling and Personalized Federated Deep Reinforcement Learning

Effective power control is key to solving the inter-user interference problem that degrades performance in cell-free massive multiple-input multiple-output (MIMO) systems. Motivated by its ability to operate online and model-free, without relying on training datasets, we leverage deep reinforcement learning (DRL) for uplink power control, aiming to maximize the guaranteed rate. We propose a fully centralized single-agent framework and two distributed schemes that employ several agents for improved scalability, leveraging prioritized experience replay to enable fast adaptation to the dynamic changes of the wireless environment. We investigate the performance of two multi-agent system architectures: (1) centralized training, decentralized execution (CTDE), where each agent forwards its RL experience to a central trainer, and (2) personalized federated learning (FedPer), where the training is performed locally at each agent, and only the base layer of the local deep neural network (DNN) model is forwarded periodically for aggregation at a server. We focus on the realistic scenario of dynamic device (de-)activation, combined with user mobility. Numerical evaluations demonstrate that the proposed FedPer with prioritized sampling achieves near-optimal rate and power performance while incurring the least amount of communication overhead.