Оцінювання ефективності великих мовних моделей для вилучення сутностей з неструктурованих документів

The object of research is arrays of unstructured documents located on public websites of rural and urban communities of Ukraine. The study is devoted to solving the problem of choosing a large language model (LLM), which is the best for applied use in solving named entity recognition (NER) problems during document processing. Modern researchers recognize that such a choice is significantly influenced by the features of the subject area and the language of document creation. However, when studying the feasibility of using LLM to solve NER problems, the features of the operation of such models are practically not taken into account. The issues of evaluating such features remain largely unexplored. A method for recognizing selected varieties of legal unstructured texts in the Ukrainian language is proposed. Unlike existing ones, this method solves the NER problem for those documents that are subject to recognition/classification. Metrics for the cost of processing input and output tokens are proposed and a methodology for evaluating the cost of using LLM is developed. Based on these results, a comparative evaluation of the application of common LLMs to solve the NER problem on Ukrainian texts that need to be recognized was conducted. According to the evaluation results, it was recognized that: (I) GPT-4o is the best in terms of accuracy and quality of processing (Precision = 0.919; Recall = 0.954; F1 = 0.936); (II) GPT-4o-mini with discounts is the best in terms of average document processing cost (0.00045 USD per document); (III) GPT-4.1-mini with discounts is the best in terms of quality/cost ratio (the indicator value is 0.938). The GPT-4.1-mini LLM is recommended as the best for applied application. The evaluation results obtained allow to significantly simplify the choice of LLM, which is advisable to use for creating information systems and technologies for processing unstructured documents created in Ukrainian.