Retrieving Operation Insights with GenAI LLM: Comparative Analysis and Workflow Enhancement

Abstract Identifying drilling risks for upcoming wells demands insights from offset wells. While some risks may lead to non-productive time (NPT), others remain latent within daily drilling reports (DDRs), which is crucial for comprehensive analysis. Manual extraction from DDRs is often time-consuming. In this paper, we systematically explore the methods to leverage the Large Language Models (LLMs) with enhanced workflows to automate insights extraction and improve risk identification accuracy. We compared LLMs customization approaches and state-of-the-art NLP models to validate LLMs efficiency in the drilling domain. We show that the adaptation of prompt optimization, Retrieval Augmented Generation (RAG) with the pre-trained LLMs before fine-tuning the model, already shows a significant improvement, achieving an accuracy ranging from 80-85%, with precision and recall surging from less than 50%, as observed from our benchmark oil and gas finetuned GPT-2 model (Marlot et al., 2023), to over 70% with our customized model. We addressed the challenge of imbalance training examples during the finetuning of the state-of-the-art GPT-2 model, which caused the underperformance of the benchmark model, particularly in addressing rare drilling event classes. This paper demonstrates LLMs able to improve the generalization and reliability of the imbalance drilling risks, with proper workflow enhancements. Subsequently, we also finetuned both on-premises LLMs (Llama2, Llama3, Mistral), and cloud LLM GPT-3.5 Turbo, which exhibits promising outcomes, achieving better performance to over 90% accuracy, and over 80% precision and recall, compared to the earlier approach using prompt and RAG. This paper contributes to advancing the application of LLMs in the oil and gas industry, for domain-specific tasks, addressing the common challenge of imbalance operation training data, and demonstrating our enhanced workflow outperforms both the benchmark finetuned GPT-2 model, pretrained GPT-4 model, and the finetuned GPT-3.5 Turbo.