ReptiLoRA: Accelerating Meta?Initialization with Low?Rank Adaptation

Meta-learning has recently gained significant attention in AI because it enables models to rapidly adapt to new tasks with limited data.One of the main goals in meta-learning is few-shot learning that makes a model to generalize to new tasks using only a small number of labeled examples.To address the challenge of few-shot learning with limited data, we propose ReptiLoRA that combines Reptile-style meta-learning with low-rank adaptation (LoRA) on large language models.Unlike model-agnostic meta-learning (MAML), which incurs high computational costs due to second-order derivatives, Reptile utilizes firstorder updates that promote fast adaptation without complex gradient computations, and LoRA further strengthens Reptile approach by introducing lightweight, memory-efficient adapters, enabling scalable and parameter-efficient meta-learning across diverse natural language processing (NLP) tasks.This approach significantly reduces memory usage and training time on models like Llama-2-7B while preserving the ability to quickly adapt to new tasks.We evaluate ReptiLoRA against a standard LoRA baseline on five summarization datasets under 0-, 1-, and 5-shot settings using identical LLaMA-2 7B architecture and recall oriented understudy for gisting evaluation (ROUGE) metrics, isolating the impact of meta-learned initialization.ReptiLoRA consistently outperforms plain LoRA especially on complex datasets like CNN/DailyMail and Multi-News.