Joint Optimization of Communication and Device Clustering for Secure Clustered Federated Learning

In this paper, a secure and communication-efficient clustered federated learning (CFL) design is investigated. In our model, several base stations (BSs) with heterogeneous task-handling capabilities and multiple users with non-independent and identically distributed (non-IID) data jointly perform CFL training using differential privacy (DP) techniques. Since each BS can process only a subset of learning tasks and has limited wireless resource blocks to allocate to users for federated learning (FL) model parameter transmission, it is necessary to jointly optimize resource block (RB) allocation and user scheduling for CFL performance optimization. Meanwhile, our considered CFL requires devices to use their limited data and FL model information to determine their task identities, which may introduce additional communication overhead. This problem is formulated as an optimization problem whose goal is to minimize the training loss of all learning tasks while considering device clustering, RB allocation, noise, and FL model transmission delay. To solve this, we propose a novel value decomposed multi-agent reinforcement learning (VD-MARL) algorithm that enables distributed BSs to independently determine their connected users, the RBs, and DP noise of the connected users but jointly minimize the training loss of all learning tasks across all BSs. Different from the existing MARL methods that assign a large penalty for invalid actions, we propose a novel penalty assignment scheme that assigns penalty depending on the number of devices that cannot meet communication constraints (e.g., delay), which can guide the MARL scheme to quickly find valid actions thus improving the convergence speed. Simulation results show that the VD-MARL can improve the convergence rate by up to 35% and the ultimate accumulated rewards by 27% compared to independent Q-learning.