An Investigation of Parameter Efficient Federated Learning with Foundation Model

Federated learning is a sub-area of distributed learning where user privacy is particularly concerned. However, the increasing scale of modern machine learning models, coupled with the resource constraints of user devices, poses significant challenges in terms of server-client communication efficiency and local model training. Various research efforts have been engaged to address them, such as model compression, pruning and parameter distillation with usually side effects of performance degradation. In this paper, we propose a novel approach that leverages a foundation model, specifically CLIP, to simultaneously achieve efficient local model training, reduced communication overhead, and superior model performance. We explore various parameter-efficient designs based on CLIP and identify the most effective strategies under extremely low-resource conditions. Our empirical evaluations provide valuable insights for future research in parameter-efficient federated learning.