SecLoRA: Efficient Privacy-Preserving LLM Tuning

Currently, the advent of ubiquitous Large Language Models (LLMs) seamlessly provides cloud-based services for Internet of Things devices with intrinsic traits. Low-Rank Adaptation (LoRA) is designed for fine-tuning large-scale models in a computation-efficient manner, yet the initial LoRA parameters delay model convergence rates and the risks of sensitive data coupled with proprietary models can not be ignored. Prior works on privacy-preserving LoRA training only focus on the privacy of local data or prediction results and sacrifice somewhat utility degradation for appropriate privacy protection. To accomplish faster training convergence rates and a higher security level, we design a pre-optimizing algorithm for local LoRA parameters, and propose an efficient scheme (SecLoRA) for secure LLMs tuning. SecLoRA leverages inner product functional encryption and matrix permutation techniques to ensure the privacy of local sensitive data and fine-tuned model parameters while also guaranteeing the reliability of prediction results to resist the tampering attacks. Convergence analysis corroborates the feasibility of local pre-optimizing algorithm. Compared with related works, we provide security analysis of SecLoRA to prove the higher level of security in terms of model privacy, prediction privacy, and prediction integrity. Extensive evaluations on numerous LLMs and public benchmarks indicate that local parameter pre-optimizing algorithm achieves a 20% 40% reduction in training time, while SecLoRA quantitatively outperforms the related works across multiple evaluation criteria.