Decentralized Federated Learning towards Communication Efficiency, Robustness, and Personalization

Decentralized federated learning emerged to eliminate the reliance on the central server and address the single point of failure and the network bottleneck in centralized federated learning. However, existing works on decentralized federated learning suffer from the following three challenges. First, the transmission of model parameters between devices results in significant bandwidth consumption and network congestion. Second, the decentralized architecture involves numerous devices, which increases the risk of poisoned behavior. Third, the data heterogeneity of devices seriously affects the model accuracy. Unfortunately, there is a lack of research that can effectively address all the challenges above. In this article, we propose a novel scheme of D ecentralized federated learning toward C ommunication E fficiency, R obustness, and P ersonalization (i.e., D-CERP). We aim at customizing personalized models for each client with lower communication and computation overhead, which can also defend against Byzantine attacks in the decentralized scenario. Specifically, we employ local sparse training with a personalized mask to better fit the heterogeneous data for each client and reduce both on-device computation overhead and cross-device communication overhead. Besides, we apply a trusted neighbor selection scheme based on multi-armed bandit by assigning rewards to high-quality submissions of each communication round, thereby improving the Byzantine robustness. In our experiments, we utilize two data partitioning methods to simulate the heterogeneity of clients in the decentralized setting and conduct exhaustive experiments on CIFAR10, CIFAR100, and Tiny-ImageNet. Experimental results demonstrate that compared to several state-of-the-art baselines, D-CERP achieves comparable personalization with a lower overhead in non-adversarial settings and provides additionally superior Byzantine robustness in adversarial settings.