Application and evaluation of RAG technology in civil aviation policy question answering

Civil aviation policies are crucial for practitioners, but the current method of manually searching for relevant policies is cumbersome and inefficient. Existing question-answering systems based on knowledge graphs or large language models are not ideal due to the lack of high-quality policy question-answering datasets and the inability to update in real-time. To this end, this paper builds a civil aviation policy question-answering system based on Retrieval-Augmented Generation (RAG). The system obtains civil aviation policy-related knowledge from multiple data sources and segments it, uses ZhipuEmbedding-3 and Chroma to create a civil aviation policy text block vector library, combines vector retrieval technology to complete the retrieval of Query text blocks, obtains the context most relevant to the Query, and then combines the prompt word function of the large language model to integrate the Query and context information in real time to generate accurate answers. Finally, we use the Ragas evaluation framework to conduct a comprehensive evaluation of the system. The results show that Qwen-max has the best overall performance in the indicators related to Faithfulness, Answer Relevancy, and Context Recall, which are 0.89, 0.75, and 0.90, respectively. The reliability and effectiveness of the system have been verified.