An Empirical Study on Large Language Models for Aerospace Question Answering

The application of knowledge-based question answering (QA) systems in the aerospace sector is crucial, providing essential support in training, fault analysis, and operational manual development. These systems enhance efficiency and safety by enabling rapid, precise answers to complex technical questions, aiding in maintaining system reliability. However, limited empirical research has focused on assessing the performance of QA systems specifically tailored for aerospace, leaving an important knowledge gap in understanding their domain-specific effectiveness. This oversight may stem from the sector's unique demands for accuracy and the technical specialization required. Addressing the challenges of building an aerospace QA system requires a model with in-depth technical comprehension, able to deliver precise and contextually relevant answers across a wide range of specialized questions. Moreover, constructing a comprehensive dataset for such complex, evolving knowledge domains presents additional hurdles. This study provides an empirical assessment of three large language models—Gemma2, LLaMA, and Phi3—on a carefully curated dataset of 300 aerospace questions. Using expert human evaluation to validate model performance, we identify each model's strengths, limitations, and practical applicability in aerospace QA. Our findings highlight that while these models show promise in answering aerospace-specific questions, their accuracy, logical structure, and depth of procedural understanding vary. These insights contribute valuable guidance for future improvements in knowledge transfer, operational support, and technical training tools within the aerospace industry.