MeMoTune: A Measure and Moment-Driven Fine-Tuning Framework for Quantized Large Language Models

Quantizing large language models (LLMs) is essential for reducing memory and computational costs in natural language processing.Existing methods combine quantization with parameter-efficient fine-tuning but often fail to meet practical performance requirements.This paper introduces MeMoTune, a novel fine-tuning framework for quantized LLMs.By employing a measure and moment approach within a low-rank approximation framework in probability measure space, MeMoTune optimizes the objective function for superior fine-tuning results.The update process is further refined through scaled gradient, enhancing convergence efficiency and noise robustness.Experiments on tasks like text generation, summarization, and understanding show MeMoTune significantly outperforms state-of-the-art methods, e.g.fine-tuning Llama2-13B on GSM8K improves accuracy by 5.5%, while fine-tuning DeBERTaV3-base on CoLA of GLUE increases Matthews correlation by 1.7%.The code is publicly available at github.com/hddyyyb/MeMoTune. LoRA AdapterC. Elizas overtime pay is $10 x 1.2 = $<<10*1.2=12>>12.Her total earnings for the first 40 hours is $10 x 40 = $<<10*40=400>>400.Her total overtime pay is $12 x 5 = $<<12*5=60>>60.Therefore, Elizas total earnings for this week is $400 + $60 = $<<400+60=460>>460.The final answer is: 460 Ground Truth: 460.