FedHiDiC: High-Dimensional Heterogeneous Data Condensation and Distillation Contrastive Learning in Federated Learning

In response to key challenges in federated learning, particularly those concerning the heterogeneity of locally distributed data which engenders client drift and the problematic occurrence of dimensionality collapse, we propose FedHiDiC: high-dimensional heterogeneous data condensation and distillation contrastive learning. A key idea of FedHiDiC is to integrate mean squared error, standard deviation, and covariance as regularization terms in local training to control parameter norm growth and bolster the capacity to handle high-dimensional, heterogeneous data. Another key idea is to utilize the similarity of representations between models to correct for local training, which reinforces intra-class compactness and inter-class distinction and reduces model variability through knowledge distillation from the global to local models, combined with contrastive strategies. Our experiments on CIFAR10 and CIFAR100 datasets show that FedHiDiC achieves significant performance improvements, surpassing FedAvg, FedProx, and MOON by 2.9% to 4.0% in accuracy, which confirms the effectiveness of FedHiDiC in tackling heterogeneity challenges.