Optimizing Federated Learning Efficiency: A Comparative Analysis of Model Compression Techniques for Communication Reduction

Abstract. The rapidly growing number of connected devices continuously produces massive amounts of heterogeneous data, posing challenges to data privacy and security when leveraging them to train the high-quality big model. Federated learning, which enables numerous users to train a global model without cooperatively exchanging data, has emerged as a viable alternative. Though achieving significant progress, existing federated learning methods still struggle with large communication volumes, especially when the local devices only have limited computing and communication capabilities. To alleviate the efficiency issue, this paper compares the effect of three compression methods in promoting the training of federated learning models, including pruning, quantization, and knowledge distillation. The findings reveal that these methods reduce resource consumption while maintaining high model performance. Combining the pruning, quantization, and knowledge distillation technology through sequential application and parameter aggregation helps balance the model size and performance. Our cascading lightweight strategy, which preserves each method's unique edge while promoting deeper collaboration between them, has been shown beneficial through extensive testing.