When Annotators Disagree: Re-evaluating Hausa Sentiment Models
This study investigates how disagreement among human annotators affects the evaluation of sentiment analysis models for Hausa, a low-resource African language. Using the Hausa subset of the AfriSenti benchmark, we examine whether evaluating models against individual annotator judgments produces different conclusions from conventional evaluation using a single reference label. The analysis covers 4,562 eligible social media posts with three distinct annotator votes and compares classical machine learning baselines with multilingual transformer models, including AfriBERTa, XLM-R, and mBERT. Model performance is assessed using macro-F1 scores, empirical vote agreement, and paired uncertainty estimates, with particular attention to unanimous and contested annotations. Results show that overall model rankings remain consistent across both evaluation metrics, although small, statistically uncertain differences emerge on posts where annotators disagree. The study also identifies 64 instances where released sentiment labels differ from the majority of individual annotator votes. These findings highlight the importance of preserving annotator perspectives, documenting annotation coverage, and accounting for uncertainty when evaluating natural language processing systems. The research contributes a reproducible evaluation audit for Hausa sentiment analysis and provides insights into more transparent and disagreement-aware evaluation practices for low-resource languages.
Authors
- Jeffrey Karau Dauda (ORCID: https://orcid.org/0009-0009-8309-7621)
- David Emmanuel
- Moussa Moustapha Moustapha
Institutions
- African Leadership University (RW)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24221110
- Primary Topic
- Sentiment Analysis and Opinion Mining
- Type
- preprint