Fed-PriLoRA: A Synergistic Framework for Private and Robust Federated Low-Rank Adaptation

Utilizing federated learning to fine-tune large language models on sensitive private data while achieving strong privacy protection without sacrificing model utility remains a fundamental challenge. Although parameter-efficient fine-tuning methods such as LoRA significantly reduce communication overhead, integrating them with differential privacy introduces issues including multiplicative noise amplification, cross-round noise accumulation, and consequent training instability, leading to severe performance degradation under strict privacy constraints. To address the above challenges, this paper proposes the Fed-PriLoRA (Private and Robust Federated Low-Rank Adaptation) framework. The framework decomposes each client’s low-rank update into separable vector pairs. It then employs structured freezing to suppress multiplicative amplification of differential privacy noise, and leverages Fisher information for noise-aware parameter importance evaluation to distinguish generalizable features from noise interference. By using Fisher information to assess parameter importance, the client locally discards low-contribution components to mitigate noise interference prior to uploading, thereby reducing communication overhead. Fed-PriLoRA achieves higher accuracy than the non-private StackLoRA baseline on both the Dolly-15K and WizardLM datasets. In terms of communication efficiency, the per-round communication overhead is reduced to just 0.051% of that incurred in full fine-tuning. Ablation experiments demonstrate that Fisher-guided pruning yields a relative accuracy increase of 39% over random pruning.