Performing Information Extraction for Mission Engineering Applications

The process of extracting structured data from unstructured and semi-structured text is manual, time consuming and error prone. Current natural language processing approaches for automating this process are difficult to verify for non-trivial and context-sensitive corpora. Large Language Models (LLMs) like ChatGPT have become a subject of considerable interest, opening a promising avenue of exploration. However, there is limited evidence on the performance of LLMs for information extraction. In this dissertation, an approach is proposed to evaluate the accuracy of Stanford OpenIE and OpenAI's ChatGPT for this purpose. This includes comparing Resource Description Framework (RDF) triples extracted by each of these semi-automated methods to hand extracted triples. For identified discrepancies and/or noteworthy extractions, qualitative indicators were collected, analyzed, and discussed. The F2 score, a measure of accuracy for each method with a weighting for recall, was calculated by combining recall and precision performance metrics. Results show that ChatGPT correctly identified manually extracted RDF triples with no statistical difference and was found to have an F2 score of 95.9%. OpenIE was found to have an F2 score of 20.7%. While LLMs still require human verification, this research has demonstrated that LLMs show an improvement over the state of the art.