FedGKD: personalized federated learning through grouping and distillation

As is well known, the objective of traditional Federated Learning (FL) is to train a global model collaboratively across multiple clients without directly accessing client data. However, traditional federated learning is frequently impeded by the heterogeneity of data, targets, and models. This work proposes a novel paradigm for federated learning, namely Federated Group Distillation (FedGKD). First, clients are grouped according to their needs and conditions. Subsequently, a knowledge distillation strategy, designated as DML, is employed to perform local distillation and global distillation on the grouped clients sequentially until the model converges. The experiments demonstrate that FedGKD is capable of effectively addressing the aforementioned three types of heterogeneous interference. Furthermore, clients can benefit from FedGKD with diverse tasks and models, ultimately achieving enhanced performance. Additionally, FedGKD is capable of reducing the load on the server to a certain extent.