Fine-Tuning Scheme for Enhancing Generalization Capability of Large Pre-Trained Models in Wireless Networks

By leveraging the parameter-efficient fine-tuning (PEFT) technology, large pre-trained models (LPMs) can effectively capture task-specific PEFT knowledge and generalize to downstream artificial intelligence (AI) tasks emerging in wireless networks. However, the inherent system heterogeneity of wireless networks and data across users brings difficulties in combining the PEFT knowledge of users. This in turn hinders the enhancement of LPM's global generalization capability. In this paper, we propose a distributed fine-tuning (DFT) scheme based on PEFT knowledge sharing, to capture the global PEFT knowledge of downstream users by aggregating heterogeneous PEFT models. In DFT, we use low-rank adaption (LoRA) model to capture the PEFT knowledge of users, where the LoRA models can be heterogeneous for different users with specific resource constraints. Then, we further design an aggregation and iterative training strategy for the heterogeneous LoRA models. The numerical results verify the effectiveness and advantages of the proposed DFT scheme in enhancing the global generalization capability of LPM in resource-heterogeneous wireless networks.