Teaching Large Language Models to Understand Jeju Island with Domain-Adaptive Pretraining

The advancement of large language models (LLMs) has led to remarkable achievements in various natural language processing (NLP) tasks. Specifically, tasks requiring a deep understanding of specific topics have significantly benefited from Domain-Adaptive Pretraining (DAPT) and Low-Rank Adaptation (LoRA) fine-tuning. This paper explores enhancing LLM performance by combining these methodologies. By integrating domain-specific knowledge through DAPT and optimizing the training process with LoRA, the model achieves higher accuracy and efficiency within specific domains. As a case study, we apply these techniques to the Llama 3 model using publicly available data about Jeju Island. Experimental results demonstrate that the adjusted model provides detailed and accurate descriptions of Jeju’s tourist attractions, cultural heritage, and natural landscapes, significantly improving the efficiency in handling specialized topics. This approach proves to be a valuable method for maximizing performance across various tasks.