Flexora: Flexible Low-Rank Adaptation for Large Language Models

Large language models (LLMs) have revolutionized artificial intelligence, but their performance on specific tasks is often limited by knowledge boundaries.While fine-tuning techniques like low-rank adaptation (LoRA) aim to address this, they can suffer from overfitting.We propose flexible low-rank adaptation (Flexora), a novel method that automatically selects the most critical layers for fine-tuning to optimize performance across diverse downstream tasks.Flexora formulates layer selection as a hyperparameter optimization problem, employs unrolled differentiation for efficient solving, and identifies the most impactful layers based on optimized hyperparameters.Extensive experiments across various pre-trained models and natural language tasks demonstrate that Flexora consistently outperforms existing baselines.We provide theoretical insights and comprehensive ablation studies to elucidate the effectiveness of Flexora.Therefore, Flexora offers a robust solution to enhance LoRA finetuning for LLMs, potentially advancing the field of adaptive language model optimization. DecoderLayer