Label Noise and the Accuracy Ceiling in Arabic Sentiment Benchmarks: A Reconciled Error Audit of HARD with Replication on ASTD and ArSAS

Arabic sentiment benchmarks are widely used to rank models, yet how much of their residual error reflects genuine model failure rather than incorrect ground-truth labels is rarely audited directly. We report a reconciled, two-annotator audit of every error (n = 401) made by the best-performing model on the Hotel Arabic Reviews Dataset (HARD), with moderate item-level agreement (Cohen’s κ = 0.41–0.42), categorizing each as label noise, genuine model error, or irreducibly ambiguous. We find that 43.4% of residual errors (49.2% of false positives, 32.6% of false negatives) are attributable to incorrect ground truth. Instantiating the random classification noise model with this measurement implies, for any classifier whose errors are independent of the label noise (a conditional assumption we flag, not a demonstrated fact), a measurable-accuracy ceiling of about 98.35% on HARD derived from a lower-bound noise estimate, leaving at most roughly 2.11 points of headroom for future work. Against that ceiling we place a five-seed comparison of four ensemble-combination rules over four Arabic transformer backbones, in which stacking’s gain over the best single model is modest and not robust: 96.40% vs. 96.24% macro-F1, nominally p = 0.023 but not significant after multiple-comparison correction, and no ensemble rule reliably beats the best backbone. Replicating the full protocol on ASTD and ArSAS shows that neither the backbone ranking nor the ensemble benefit generalizes to informal-register data. We release the reconciled annotations, protocol, and evaluation code.

Authors

Institutions

Publication Details

Journal
Applied Sciences
Published
2026-11-17
DOI
https://doi.org/10.3390/app162211382
Primary Topic
Sentiment Analysis and Opinion Mining
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Label Noise and the Accuracy Ceiling in Arabic Sentiment Benchmarks: A Reconciled Error Audit of HARD with Replication on ASTD and ArSAS

Mohammad O. Alhawarat, Khalil M. Abdelnaby, Mohammed A. F. Al-Husainy, Ayman J. Alnsour
2 links
Applied Sciences
Sentiment Analysis and Opinion Mining
article

Label Noise and the Accuracy Ceiling in Arabic Sentiment Benchmarks: A Reconciled Error Audit of HARD with Replication on ASTD and ArSAS

Mohammad O. Alhawarat, Khalil M. Abdelnaby, Mohammed A. F. Al-Husainy, Ayman J. Alnsour
article en

Abstract

Arabic sentiment benchmarks are widely used to rank models, yet how much of their residual error reflects genuine model failure rather than incorrect ground-truth labels is rarely audited directly. We report a reconciled, two-annotator audit of every error (n = 401) made by the best-performing model on the Hotel Arabic Reviews Dataset (HARD), with moderate item-level agreement (Cohen’s κ = 0.41–0.42), categorizing each as label noise, genuine model error, or irreducibly ambiguous. We find that 43.4% of residual errors (49.2% of false positives, 32.6% of false negatives) are attributable to incorrect ground truth. Instantiating the random classification noise model with this measurement implies, for any classifier whose errors are independent of the label noise (a conditional assumption we flag, not a demonstrated fact), a measurable-accuracy ceiling of about 98.35% on HARD derived from a lower-bound noise estimate, leaving at most roughly 2.11 points of headroom for future work. Against that ceiling we place a five-seed comparison of four ensemble-combination rules over four Arabic transformer backbones, in which stacking’s gain over the best single model is modest and not robust: 96.40% vs. 96.24% macro-F1, nominally p = 0.023 but not significant after multiple-comparison correction, and no ensemble rule reliably beats the best backbone. Replicating the full protocol on ASTD and ArSAS shows that neither the backbone ranking nor the ensemble benefit generalizes to informal-register data. We release the reconciled annotations, protocol, and evaluation code.

Applied SciencesVol. 16(22)
Al-Ahliyya Amman University (JO), Al-Azhar University (EG), Aqaba University of Technology (JO)
Sentiment Analysis and Opinion Mining
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon