Decentralized Low-Rank Fine-Tuning of Large Language Models

While parameter-efficient fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) offer computationally efficient adaptations of Large Language Models (LLMs), their practical deployment often assumes centralized data and training environments.However, realworld scenarios frequently involve distributed, privacy-sensitive datasets that require decentralized solutions.Federated learning (FL) addresses data privacy by coordinating model updates across clients without sharing raw data.While most federated fine-tuning methods adopt centralized FL, which relies on a parameter server for aggregating model updates-introducing potential bottlenecks and communication constraints-decentralized FL enables direct peer-to-peer communication among clients, bypassing the need for a server as an intermediary.Despite its advantages, decentralized fine-tuning for LLMs remains largely unexplored in the literature.To address this gap, we introduce Dec-LoRA, a decentralized fine-tuning algorithm based on LoRA.We conduct extensive experiments using BERT and LLaMA-2 models to benchmark Dec-LoRA against centralized LoRA and several other popular PEFT approaches in decentralized settings.Our results demonstrate that Dec-LoRA consistently achieves performance on par with centralized LoRA under various conditions, including data heterogeneity and quantization constraints.These findings highlight the potential of Dec-LoRA for scalable LLM fine-tuning in decentralized environments.