Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning

Existing continual learning works explored strategies like memory replay, regularization, and parameter isolation, but little analysis was conducted on the optimization behavior of LLMs’ continual fine-tuning. In this work, we investigate the geometric connections of different minima along the continual LLM fine-tuning trajectories. We validate this phenomenon on LLMs and propose a new method called Interpolation-based LoRA (I-LoRA). I-LoRA can strike a balance between plasticity and stability through parameter interpolation, which constructs a dual-memory experience replay framework based on LoRA. Experiments on eight domain-specific benchmarks demonstrate that I-LoRA consistently shows significant improvement over previous approaches with up to 11% performance gains. Our code is available at https://anonymous.4open.science/r/LLMCL-3823.