Impact of Low Rank Adaptation with Parameter Efficient Fine-Tuning Techniques on Transformer Models

Transformer model has fundamentally transformed natural language processing with its self-attention mechanism, enabling efficient handling of long-range dependencies in text. Unlike traditional models like Recurrent Neural Networks and Convolutional Neural Networks, transformers process sequences more effectively through an encoder-decoder structure that incorporates multi-head self-attention and position-wise feedforward networks. Artificial intelligence is becoming an essential part to perform tasks more efficiently than humans or traditional machines, excelling in understanding humans, computers, and programming languages. Large Language Models (LLMs) have emerged as key tools in Natural Language Processing (NLP). This paper explores the rank analysis of Low-Rank Adaptation (LoRA) using Parameter Efficient Fine Tuning (PEFT) techniques for transformer models. By incorporating low-rank decomposition into model layers, the objective of LoRA is to enhance fine-tuning efficiency and reduce computational requirements. We evaluated LoRA on a text summarization data set using various metrics to measure performance. Our findings reveal that LoRA not only maintains high accuracy in summarization tasks but also significantly improves computational efficiency, making advanced AI research more accessible to those with limited resources. Our study highlights LoRA's potential to optimize LLM's paving the way for more efficient and inclusive AI advancements.