Toward Scalable Human-in-the-Loop Annotation Error Detection with Label Noise-Aware Training

Annotation errors in datasets have a substantial impact on the performance of AI systems, both during training and evaluation. Although accurate labeling is crucial, human-in-the-loop methods for error detection face limitations due to scalability challenges and susceptibility to fatigue. In this work, we investigate Annotation Error Detection (AED) techniques with an emphasis on the effects of noise-robust training strategies. We explore how label noise affects AED performance and examine the effectiveness of robust regularization and robust loss based training in mitigating the negative effects of noisy labels. Our results show that standard models trained with noisy data experience significant performance drop; however, simple techniques from robust regularization improve AED performance by 20% to 45%. Our study underscores the importance of integrating noise-robust methods into existing AED systems for improving overall dataset quality.