Expediting Federated Learning on Non-IID Data by Maximizing Communication Channel Utilization

Federated learning (FL) is at the core of intelligent Internet architecture. It allows clients to jointly train a model without direct data sharing. In such a process, clients and the central server share information through communication channels formed by parameters. However, the non-iid training data in clients significantly impacts global model convergence and brings difficulties for the evaluation of local contributions. Most of existing studies try to expand the communication channel by improving consistency with variance reduction or regularization, but such methods neglect an important factor, i.e., channel utilization, hence their capability for sharing information is under-utilized. Moreover, the issue of contribution evaluation is still unsolved. In this paper, we simultaneously solve the former two challenges (i.e., model convergence and contribution evaluation) by modeling the indirect data sharing of FL as a problem of information communication. We prove that FL with non-iid data forms noisy communication channels, which have limited capability for information transmission, i.e., limited channel capacity. The main factor in deciding the channel capacity is the Gradient Signal to Noise Ratio (GSNR). Through analyzing GSNR, we further prove that channel capacity can be reached by optimal local updates and propose a method FedGSNR to calculate it, which allows us to maximize channel utilization in FL, leading to faster model convergence. Moreover, as the contribution of the local dataset depends on the amount of provided information, the derived GSNR allows the server to accurately evaluate the contributions of different clients (i.e., the quality of local datasets).