When Annotators Disagree: Re-evaluating Hausa Sentiment Models

This study investigates how disagreement among human annotators affects the evaluation of sentiment analysis models for Hausa, a low-resource African language. Using the Hausa subset of the AfriSenti benchmark, we examine whether evaluating models against individual annotator judgments produces different conclusions from conventional evaluation using a single reference label. The analysis covers 4,562 eligible social media posts with three distinct annotator votes and compares classical machine learning baselines with multilingual transformer models, including AfriBERTa, XLM-R, and mBERT. Model performance is assessed using macro-F1 scores, empirical vote agreement, and paired uncertainty estimates, with particular attention to unanimous and contested annotations. Results show that overall model rankings remain consistent across both evaluation metrics, although small, statistically uncertain differences emerge on posts where annotators disagree. The study also identifies 64 instances where released sentiment labels differ from the majority of individual annotator votes. These findings highlight the importance of preserving annotator perspectives, documenting annotation coverage, and accounting for uncertainty when evaluating natural language processing systems. The research contributes a reproducible evaluation audit for Hausa sentiment analysis and provides insights into more transparent and disagreement-aware evaluation practices for low-resource languages.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24221110
Primary Topic
Sentiment Analysis and Opinion Mining
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When Annotators Disagree: Re-evaluating Hausa Sentiment Models

Jeffrey Karau Dauda, David Emmanuel, Moussa Moustapha Moustapha
Zenodo (CERN European Organization for Nuclear Research)
Sentiment Analysis and Opinion Mining
preprint

When Annotators Disagree: Re-evaluating Hausa Sentiment Models

Jeffrey Karau Dauda, David Emmanuel, Moussa Moustapha Moustapha
preprint en

Abstract

This study investigates how disagreement among human annotators affects the evaluation of sentiment analysis models for Hausa, a low-resource African language. Using the Hausa subset of the AfriSenti benchmark, we examine whether evaluating models against individual annotator judgments produces different conclusions from conventional evaluation using a single reference label. The analysis covers 4,562 eligible social media posts with three distinct annotator votes and compares classical machine learning baselines with multilingual transformer models, including AfriBERTa, XLM-R, and mBERT. Model performance is assessed using macro-F1 scores, empirical vote agreement, and paired uncertainty estimates, with particular attention to unanimous and contested annotations. Results show that overall model rankings remain consistent across both evaluation metrics, although small, statistically uncertain differences emerge on posts where annotators disagree. The study also identifies 64 instances where released sentiment labels differ from the majority of individual annotator votes. These findings highlight the importance of preserving annotator perspectives, documenting annotation coverage, and accounting for uncertainty when evaluating natural language processing systems. The research contributes a reproducible evaluation audit for Hausa sentiment analysis and provides insights into more transparent and disagreement-aware evaluation practices for low-resource languages.

Zenodo (CERN European Organization for Nuclear Research)
African Leadership University (RW)
Sentiment Analysis and Opinion Mining
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon