Run LoRA Run: Faster and Lighter LoRA Implementations

LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers.This technique is used for fine-tuning and even training large transformer models from scratch.This paper presents the RunLoRA framework for efficient implementations of LoRA, which significantly improves the speed of neural network training and fine-tuning with low-rank adapters.The proposed implementation optimizes the computation of LoRA operations based on the shape of the corresponding linear layer weights, the input dimensions, and the LoRA rank by selecting the best forward and backward computation graphs based on FLOPs and time estimations.This results in faster training without sacrificing accuracy.The experimental results show a speedup ranging from 10% to 28% on various transformer models.