An Approach to Information Extraction from Texts of a Limited Subject Domain Based on a Chain of Large Language Models

The paper considers the approach for extracting the information from texts of limited domains of knowledge based on a chain of neural language models.The task is represented in the form of three subtasks solved sequentially:(1) term extraction and classification; (2) coreference resolution;(3) extraction of relations of entities named with the terms.The dataset was based on texts on computational linguistics from the Habr forum.In the markup for term classification and relation extraction, 17 classes of terms and 51 relations were used in accordance with the ontology of computational linguistics.Prompt chain-based methods were used to apply LLMs, where each next query to the LLM is based on the results of the previous step.Six types of prompt templates were developed: for extracting, classifying, verifying terms, extracting coreferential relations, relations specified by the ontology, and a specialized template for relations linking entities of the same class.Sentence-BERT, GPT-4 and Mistral-based models were used at different steps of the study; a comparison with the SFT approach (ruRoBERTa) was made; hybrid approaches that have shown the best results were also developed.For term extraction and classification, F1=0.77 was obtained, for coreference resolution-F1=0.897,and for relation extraction-F1=0.847.