Error Candidate Cross-Validation for Efficient Annotation Error Detection

Annotated corpora play a crucial role in supervised machine learning, particularly in natural language processing (NLP); however, labels assigned by humans are not always reliable. Recent studies have predominantly employed the k-fold cross-validation method, which involves training machine learning models to predict data labels and detect annotation errors. Although increasing the number of folds or performing multiple validation cycles with different training subsets can improve error detection accuracy, these methods often result in considerable computational cost. In this study, we propose the Annotation Error Candidate Cross-Validation method, which selectively uses data with low error probabilities solely for training, while repeatedly evaluating high-error-probability data with various trained models. Our experimental results demonstrate that the proposed method accurately identifies deliberately inserted label errors in NLP tasks. Furthermore, to support the re-annotation process, we investigate the extent to which error probabilities should be reviewed to optimize annotation efficiency.