The methodology of multi-criteria evaluation of text markup models based on inconsistent expert markup

A wide class of natural language processing tasks is solved using markup.At the moment, the vast majority of models and datasets rely on a simple markup structure containing only fragments and labels.Moreover, simple classification metrics such as F1, Precision, Recall are used to evaluate the model's accuracy.The problem with such metrics is that they do not take into account all aspects of the markup structure and that they are applicable only under the assumption of the existence of an ideal markup.This paper proposes a more general and universal markup structure that allows solving complex problems and builds a methodology for multi-criteria evaluation of text markup models based on inconsistent expert markup.After that, the application of the constructed method is considered to assess the quality of the model obtained within the winning algorithm of the "READ//ABLE" competition, which focused on building an effective essay markup system.The results demonstrate that the new markup structure and evaluation approach provides a more comprehensive and accurate assessment of model performance, addressing the limitations of traditional metrics by accounting for complex markup scenarios and expert inconsistencies.