Efficient Domain Continual pretraining by Mitigating the Stability Gap

Continual pretraining enables Large Language Models (LLMs) to adapt to specialized domains like medicine and law.However, we observe a consistent phenomenon across different model sizes and domains: a temporary performance drop at the start of the continual pretraining process, followed by a performance recovery phase.To gain a deeper understanding of this issue, we use the stability gap-a concept adapted from the visual domain-which explains this initial drop arises from instability in the model's general abilities.We validate this hypothesis through a series of experiments.To address this initial instability and enhance LLM performance within a fixed compute budget, we propose a training strategy that mitigates instability by increasing the number of epochs, alongside two data sampling strategies targeting data domain relevance and corpus distribution.We conduct experiments on Llamafamily models to validate the effectiveness of our strategies for continual pretraining and instruction tuning in medical and legal domains.Our strategies improve the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% using only 40% of the original training budget, while also enhancing general task performance without causing forgetting.Furthermore, we apply our strategies to continually pre-train and instruction-tune the Llama-3-8B model.The resulting model, Llama-3-Physician 1 , achieves the best medical performance among open-source models and rivals GPT-4 on specific tasks.