A Survey of Low-Rank Adaptation Techniques

The advent of the Transformer architecture has spurred the development of Large Language Models (LLMs), yet fine-tuning these models for downstream tasks poses challenges due to high computational costs. Low-Rank Adaptation(LoRA) has emerged as a leading parameter-efficient fine-tuning (PEFT) method, LoRA is based on the Low Intrinsic Dimensionality Hypothesis, reducing trainable parameters through low-rank matrices decomposition while maintaining or enhancing performance. This paper comprehensively reviews LoRA, covering its low-rank adaptation concept, technical optimizations, architectural innovations, existing challenges and future prospective.