Measuring Label Ambiguity in Subjective Tasks using Predictive Uncertainty Estimation

Human annotations in natural language corpora vary due to differing human perspectives.This is especially prevalent in subjective tasks.In these datasets, certain data samples, i.e. annotatable instances, are more prone to label variation and can be indicated as ambiguous.This paper investigates methodologies for quantifying such label ambiguity by leveraging uncertainty estimation techniques when fine-tuning transformer-based models.We conducted experiments on three tasks characterized by subjective content and inherent label ambiguity: classifying sentiment, emotions and hate speech.The selected datasets include multiannotator labels, which we use to derive a label ambiguity score for each data sample.This score is the entropy of the empirical probability distribution of annotator labels.The results indicate that uncertainty estimation techniques can measure label ambiguity to some extent.Deep Ensembles consistently outperform other techniques, increasing the correlation coefficients between model uncertainty and annotator disagreement, but the observed correlations are low.When comparing the annotator label distributions with the predicted class distributions, we see that Label Smoothing is able to notably reduce this difference, however a discrepancy still exists.This suggests that uncertainty estimation techniques improve the quantification of label ambiguity, however their ability remains limited, highlighting the need for further research 1 .