A Retrieval-Augmented Generation-Based Method for Aviation Accident Data Analysis

In response to the growing demand for more intelligent and efficient methods of analyzing aviation accidents, this paper explores an AI-based approach to data mining and utilization in aviation accident investigation reports. Based on the framework of Retrieval-Augmented Generation (RAG), we constructed a benchmark dataset of aviation accident reports, we also built an external knowledge base from the Air Transport Association Specification 100 (ATA100) and enhanced the content with components in Aircraft Manual Maintenance (AMM). Experiments of sensitive factors were conducted to find out the most influential factors of RAG. The results show that the LLM, the retrieved number of external knowledge segments (top-k), and especially the embedding model have the greatest influence on RAG. The optimal factors reached scores of 0.75 in recall coverage rate, 0.74 in two-digit codes accuracy and 0.53 in four-digit codes accuracy. Based on the optimal sensitive factors, we conducted experiment of external knowledge enhancement. The results indicate that a more comprehensive external knowledge base can benefit both LLM and embedding model as recall coverage rate improved 8%, two-digit codes accuracy improved 11%, and four-digit codes improved 16%. The outcome of our work indicates that the RAG-base analysis method can significantly enhance the efficiency and accuracy of aviation knowledge extraction task while being highly automated and self-iterative, demonstrating its potential for future applications in the aviation industry.