FedRD: Personalized Federated Learning via Representation Distillation

Existing Federated Learning (FL) methods are mostly based on the model-averaging scheme, in which the central server averages local model parameters to obtain the latest global model. However, this traditional scheme suffers serious accuracy degradation in non-i.i.d settings due to the update bias. What's worse, transmitting model parameters leads to high communication overheads and privacy issues. Inspired by the dataset distillation (DD) methods, we propose a novel DD-based FL scheme called FedRD for addressing aforementioned problems. In FedRD, the whole learning model consists of the local projection model and the global task model. The clients firstly use the personalized projection model to transform local dataset into low-dimensional representations. Then, the synthetic set of data representations is learned via distribution matching. By sharing synthetic representations, clients can transmit the information of local dataset to the central server in a privacy-preserving and communication-efficient way. In the server side, the global task model is trained over the aggregation of synthetic representations. It enables FedRD to avoid the update bias in model-averaging schemes caused by data heterogeneity. Experiments are conducted over four image classification datasets, and the results demonstrate that FedRD outperforms compared methods significantly in terms of model accuracy and communication overhead.