Dialect-Level Generalization of Arabic Sentiment Classifiers: An Exact-Duplicate-Free, Uncertainty-Aware Evaluation of Encoders and LLMs

Preprint and complete reproduction archive for "Dialect-Level Generalization of Arabic Sentiment Classifiers: An Exact-Duplicate-Free, Uncertainty-Aware Evaluation of Encoders and LLMs." AbstractHow well do Arabic sentiment classifiers generalize across dialects, and how should this be measured? We call a classifier robust across dialects if its macro-F1 is both high and even, summarized by harmonic-mean (HM-F1) and worst-dialect F1 with hierarchical bootstrap intervals over tweets and seeds. We evaluate 11 fine-tuned encoders and four instruction-tuned LLMs on a deduplicated, leak-free version of ArSarcasm (MSA, Egyptian, Levantine and Gulf Arabic) and replicate the transfer experiments on ArSarcasm-v2. The official ArSarcasm splits share 44 tweets between training and test, but the resulting inflation is small (-0.005 to +0.004 HM-F1 across four core encoders). Under pooled training MARBERTv2 is best (HM-F1 0.870), but differences among the 7 strongest Arabic encoders, and to the best 8-shot LLM (Fanar-1-9B-Instruct, 0.849), lie within uncertainty, and Levantine is the hardest dialect for every encoder. With training size matched, leaving the target dialect out or training on MSA only costs Arabic encoders no detectable HM-F1: losses above about 0.045 are excluded, although single-dialect effects below roughly 0.07-0.10 cannot be resolved, and adding in-dialect tweets yields no statistically detectable gain. This replicates on ArSarcasm-v2; the leave-one-dialect-out loss of the multilingual XLM-R does not. With tokenizer fixed, dialect-only and classical pretraining are less robust than mixed or MSA pretraining. A variance-based robustness index, on accuracy or F1, ranks a majority classifier among the most robust of 79 systems and should not be used. Automatic dialect identification agrees with only 0.56 of the gold dialect labels, yet conclusions and encoder tiers are unchanged on the agreeing subset. Archive Contents- main.pdf: Full 24-page preprint manuscript (CC BY 4.0).- arabic-dialect-robustness-zenodo.zip: Complete reproduction archive containing LaTeX source files, all 21 tables and 5 vector figures, sanitized split metadata (internal row indices, split assignments and labels with raw tweet text and Twitter/X identifiers removed), multi-seed prediction arrays and Python verification scripts (MIT License for code; CC BY 4.0 for derived split metadata).- Code Mirror: https://github.com/Hamza-Nayab/arabic-dialect-robustness

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24326844
Primary Topic
Sentiment Analysis and Opinion Mining
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Dialect-Level Generalization of Arabic Sentiment Classifiers: An Exact-Duplicate-Free, Uncertainty-Aware Evaluation of Encoders and LLMs

Malik M. H. Nayab, Syed Ghazi Raza, Baseera Bibi
Zenodo (CERN European Organization for Nuclear Research)
Sentiment Analysis and Opinion Mining
preprint

Dialect-Level Generalization of Arabic Sentiment Classifiers: An Exact-Duplicate-Free, Uncertainty-Aware Evaluation of Encoders and LLMs

Malik M. H. Nayab, Syed Ghazi Raza, Baseera Bibi
preprint en

Abstract

Preprint and complete reproduction archive for "Dialect-Level Generalization of Arabic Sentiment Classifiers: An Exact-Duplicate-Free, Uncertainty-Aware Evaluation of Encoders and LLMs." AbstractHow well do Arabic sentiment classifiers generalize across dialects, and how should this be measured? We call a classifier robust across dialects if its macro-F1 is both high and even, summarized by harmonic-mean (HM-F1) and worst-dialect F1 with hierarchical bootstrap intervals over tweets and seeds. We evaluate 11 fine-tuned encoders and four instruction-tuned LLMs on a deduplicated, leak-free version of ArSarcasm (MSA, Egyptian, Levantine and Gulf Arabic) and replicate the transfer experiments on ArSarcasm-v2. The official ArSarcasm splits share 44 tweets between training and test, but the resulting inflation is small (-0.005 to +0.004 HM-F1 across four core encoders). Under pooled training MARBERTv2 is best (HM-F1 0.870), but differences among the 7 strongest Arabic encoders, and to the best 8-shot LLM (Fanar-1-9B-Instruct, 0.849), lie within uncertainty, and Levantine is the hardest dialect for every encoder. With training size matched, leaving the target dialect out or training on MSA only costs Arabic encoders no detectable HM-F1: losses above about 0.045 are excluded, although single-dialect effects below roughly 0.07-0.10 cannot be resolved, and adding in-dialect tweets yields no statistically detectable gain. This replicates on ArSarcasm-v2; the leave-one-dialect-out loss of the multilingual XLM-R does not. With tokenizer fixed, dialect-only and classical pretraining are less robust than mixed or MSA pretraining. A variance-based robustness index, on accuracy or F1, ranks a majority classifier among the most robust of 79 systems and should not be used. Automatic dialect identification agrees with only 0.56 of the gold dialect labels, yet conclusions and encoder tiers are unchanged on the agreeing subset. Archive Contents- main.pdf: Full 24-page preprint manuscript (CC BY 4.0).- arabic-dialect-robustness-zenodo.zip: Complete reproduction archive containing LaTeX source files, all 21 tables and 5 vector figures, sanitized split metadata (internal row indices, split assignments and labels with raw tweet text and Twitter/X identifiers removed), multi-seed prediction arrays and Python verification scripts (MIT License for code; CC BY 4.0 for derived split metadata).- Code Mirror: https://github.com/Hamza-Nayab/arabic-dialect-robustness

Zenodo (CERN European Organization for Nuclear Research)
COMSATS University Islamabad (PK)
Sentiment Analysis and Opinion Mining
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon