Application of Large Language Model and In-Context Learning for Aviation Safety Prediction

The analysis of aviation safety reports using natural language processing (NLP) techniques is well-explored, but leveraging large language models (LLMs) in this domain remains under investigation. Recent studies have shown that LLMs, equipped with tens of billions of parameters and pre-trained on massive and diverse datasets, have the ability for in-context learning (ICL). This capability allows them to learn from prompts consisting of specific instructions and a few labeled examples without requiring model fine-tuning. This paper explores the use of LLMs to classify aviation safety reports through a case study focusing on domain-specific data, utilizing ICL and prompt engineering. Using BM25 to select eight examples for the prompt resulted in the highest performance, with an accuracy of 80.24% and an F1-score of 84.15%, demonstrating the LLM's generalization ability on domain-specific data through an engineered prompt.