Tailoring Large Language Models for Drilling Applications: A Comparative Study of Retrieval-Augmented Generation and Fine-Tuning

Abstract The Fourth Industrial Revolution (4IR) is defined by its pure data-driven nature, leading to advancements in computing, sensors, and machine learning to enhance operational efficiency through data-driven approaches. However, a significant amount of data remains underutilized due to its complex and unstructured formats. It is estimated that 50% of large enterprises manage at least 5 Petabytes (PB) of data, with 80% being unstructured (Mcdowell, 2023). Some examples of these datasets in the drilling industry are derived from daily operation remarks, engineering programs, and domain literature, all containing valuable insights. While significant effort is being made in the drilling industry through Artificial Intelligence (AI) systems, these models often depend on data gathered from over 80,000 sensors deployed on drilling platforms (Patel, 2024), limiting them to quantitative, single-modality datasets. Emerging technologies in Natural Language Processing (NLP), and particularly Large Language Models (LLMs) offer promising solutions to processing unstructured data, thereby unlocking additional insights from untapped resources. Nonetheless, developing domain-specific language models from scratch is a challenging endeavor. For reference, it is estimated that the smaller 175 billion parameters GPT-3 was trained using 1024 Graphics Processing Units (GPUs) for 34 days (Narayanan et al., 2021). While the larger 1.8 trillion parameters GPT-4 was trained with approximately 25,000 GPUs over 90 days (Treiber, 2023). This paper demonstrates the potential of adapting current Generative Pre-trained Transformer models (GPT) for drilling applications. The first objective is to improve drilling knowledge of base GPT models by analyzing the effectiveness of Retrieval Augmented Generation (RAG), and Low-Rank Adaptive (LoRA) fine-tuning. The second objective is to utilize LoRA fine-tuning for Lost Time Incident (LTI) prediction using a curated dataset from operational reports. The approaches highlighted in this paper will showcase how LLMs can be tailored to the drilling domain for various tasks, cost-effectively and without extensive computational resources.