Evaluating Knowledge Loss During Continued Pretraining

Large Language Models (LLMs) have demonstrated remarkable performance across various tasks, yet their adaptation to domain-specific applications via continued pretraining remains resource-intensive. Small Language Models (SLMs) like Gemma, Phi, and LLaMA offer a cost-effective alternative, enabling efficient fine-tuning on proprietary datasets. However, continued pretraining introduces catastrophic forgetting (CF), where models lose prior knowledge while acquiring new information, posing a significant challenge for enterprise applications.This study examines the impact of continued pretraining on knowledge retention, output consistency, and model reliability. Our findings reveal that CF leads to substantial knowledge degradation and unintended repetition, limiting model adaptability. Despite ongoing research since McCloskey and Cohen’s 1989 study, CF remains a fundamental constraint in deep learning. This work highlights the need for effective mitigation strategies to preserve prior knowledge while ensuring seamless adaptation, contributing to more resilient and dependable language models.