What Is in a WER? Reproducible Evaluation of Whisper for Sudanese Arabic

Published word error rates (WER) for dialectal Arabic speech recognition are hard to compare, because text normalisation, decoding settings and data splits often go unreported. For practitioners who choose a model for real users, this matters. We re-evaluate Whisper on Sudanese Arabic with a fully specified protocol. (1) An independent run of a public finetuned checkpoint reproduces its reported score (67.38 vs. 67.7 WER). (2) Arabic orthographic normalisation (alef, ya, ta marbuta and the conjunction wa) lowers cross-corpus WER by 10.8–12.7 points for models fine-tuned on the open Sudanese corpus, against 4.9–7.1 points for zero-shot models. Much of the apparent cross-corpus error is spelling convention, not recognition. (3) Zero-shot Whisper large-v3 matches the fine-tuned small model on WER, though not on CER. (4) On a new speakerdisjoint split, LoRA adaptation helps unseen speakers of the same corpus but brings no significant gain on a second Sudanese test set. (5) A widely used mirror of the corpus pairs audio with the wrong transcripts: only 85 of 3,547 rows are aligned. We release the split, the normaliser, the scoring code and an alignment checker, and we turn these findings into an evaluation checklist for low-resource dialect ASR.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24355117
Primary Topic
Speech Recognition and Synthesis
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

What Is in a WER? Reproducible Evaluation of Whisper for Sudanese Arabic

Saifuddin Ahmed
Zenodo (CERN European Organization for Nuclear Research)
Speech Recognition and Synthesis
preprint

What Is in a WER? Reproducible Evaluation of Whisper for Sudanese Arabic

Saifuddin Ahmed
preprint en

Abstract

Published word error rates (WER) for dialectal Arabic speech recognition are hard to compare, because text normalisation, decoding settings and data splits often go unreported. For practitioners who choose a model for real users, this matters. We re-evaluate Whisper on Sudanese Arabic with a fully specified protocol. (1) An independent run of a public finetuned checkpoint reproduces its reported score (67.38 vs. 67.7 WER). (2) Arabic orthographic normalisation (alef, ya, ta marbuta and the conjunction wa) lowers cross-corpus WER by 10.8–12.7 points for models fine-tuned on the open Sudanese corpus, against 4.9–7.1 points for zero-shot models. Much of the apparent cross-corpus error is spelling convention, not recognition. (3) Zero-shot Whisper large-v3 matches the fine-tuned small model on WER, though not on CER. (4) On a new speakerdisjoint split, LoRA adaptation helps unseen speakers of the same corpus but brings no significant gain on a second Sudanese test set. (5) A widely used mirror of the corpus pairs audio with the wrong transcripts: only 85 of 3,547 rows are aligned. We release the split, the normaliser, the scoring code and an alignment checker, and we turn these findings into an evaluation checklist for low-resource dialect ASR.

Zenodo (CERN European Organization for Nuclear Research)
Speech Recognition and Synthesis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon