Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness.Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks and transfer knowledge due to its lowrank nature and reliance on explicit parameter constraints.We propose GORP (Gradient LOw Rank Projection) for Continual Learning, a novel training strategy that overcomes these limitations by synergistically combining full and low-rank parameters and jointly updating within a unified low-rank gradient subspace.GORP expands the optimization space while preserving efficiency and mitigating catastrophic forgetting.Extensive experiments on continual learning benchmarks demonstrate GORP's superior performance compared to existing state-of-the-art approaches.Code is available at https://github.com/Wcxwcxw/GORP.