Improving LLM Performance in Aerospace NER Task: Entity-Adaptive Data Augmentation and Fine-Tuning Strategies

The shift from traditional document-based practices to Model-Based Systems Engineering (MBSE) is anticipated to enhance systems engineering by introducing benefits including better integration and collaboration across system components, essential for complex system development. As MBSE adoption grows, so does the need for effective Named Entity Recognition (NER) methods capable of extracting machine-readable entities from natural language resources. NER plays an important role in transforming unstructured data into structured formats by identifying and classifying entities. Large Language Model (LLM)-based NER in particular has shown strong potential due to its advanced contextual understanding capabilities. However, fine-tuning LLMs requires large volumes of annotated data, and manual labeling process is both time-consuming and error-prone. These challenges are further exacerbated in specialized domains such as aerospace and medicine, where data is often confidential, sensitive and difficult to access. To address these challenges, this study proposes an entity-adaptive data augmentation strategy designed to generate synthetic training data tailored to the characteristics of specific entity types. By systematically varying synonym replacement rates and optimizing them per entity type, the proposed method effectively improved the performance of fine-tuned LLMs in an aerospace domain-specific NER task, using a limited dataset of only 300 aerospace requirement sentences. Notably, it improves NER performance without requiring additional manual annotation. Overall, the results demonstrate that entity-adaptive augmentation is a scalable and effective solution for enhancing domain-specific NER in data-scarce settings.