Evaluating Annotation Consistency in Offensive Language Detection: A Data Analytics Approach on the TweetEval Dataset

Abstract— Most machine learning models are not only highly dependent on difficult datasets but also on the quality of labeled data they are trained on, especially for offensive content detection. In this paper, we study the TweetEval dataset to provide a comparison of its ground truth with manually annotated labels; inter-annotator agreements are applied here as a metric for assessing the consistency of annotation. Cohen’s Kappa coefficient is used to quantify how much each pair of annotators agreed and where they differed. In-depth examination of missed classifications demonstrates other difficulties with manual labelling: subjective interpretation, context dependency, and annotator bias. The in- sights gathered demonstrate how manual annotation can have positive and negative effects on further model training practices, highlighting the importance of standardized annotation guidelines. In their actions, the findings contribute to enhancing offensive content detection models by advocating dataset reliability and the reduction of inconsistencies in labeling. Index Terms—TweetEval Dataset,Annotation Consistency, Inter- Annotator Agreement,Cohen’s Kappa, Fleiss’ Kappa,Dataset Re- liability, Text Classification,Natural Language Processing (NLP), Offensive Language Detection, Hybrid Models,Annotator Bias