Construction and validation evaluation of a high-quality dataset in the national defense mobilization domain based on large language models

With the advancement of natural language processing technology, the application scope of large language models has been expanding. However, in the domain of national defense mobilization, the application of large language models is hindered by the scarcity of high-quality datasets, making it challenging to implement these models in practical scenarios. To address this issue, this paper proposes a dataset construction method based on large language models that obviates the need for manual writing by leveraging the models to generate question-answer pairs. During the data generation phase, the stability of question-answer pair generation is enhanced through in-context learning and input priority strategies. In the data filtering phase, a data quality assessment mechanism is established by evaluating the fidelity to the original text and semantic quality, which allows for the identification and exclusion of low-quality and anomalous data, ensuring the reliability of the question-answer pairs. Experimental results demonstrate that this method ensures both the quality and efficiency of dataset construction and the performance of the model fine-tuned on this dataset is significantly improved in the question-answering tasks within the national defense mobilization domain. The research is expected to provide a reference for the application of LLMs in other domains.