Assessing Inter-Annotator Agreement On Argumentative Markup

The article discusses the issues of argumentation analysis, argumentative annotation of texts and assessment of inter-annotator agreement. Argumentation is understood as an ordered set of arguments used to confirm or refute certain theses. An argument includes one or more premises and a conclusion, connected by an argumentative relation. Each argument belongs to a specific type or scheme according to the classification of arguments proposed by D. Walton. Existing methods for assessing agreement between annotators are poorly suited for comparing argumentative text markups. The work proposes a new approach in which comparison is carried out at three levels of argumentative marking: by statements, by relations and by schemes. The study was conducted on two text corpora with argumentative annotation, created using the ArgNetBank Studio web platform. The corpora included texts of different genres related to the field of scientific communication. An analysis of typical problems of annotators during marking, leading to discrepancies, was carried out. Research has shown significant differences in agreement between annotators at all levels of argumentative text marking, which is explained by the complex structure of argumentation and high variability in the presentation of reasoning. The ways are suggested to further improve the comparison algorithm for a more objective assessment of inter-annotator agreement.