Beyond Character Error Rate: Evaluating Diacritic Fidelity and Post-Processing Effects in Classical Arabic OCR

Abstract Classical Arabic books print diacritics selectively, and the marks can change meaning. Faithful OCR of these books must copy the marks the page shows, adding none and dropping none. Arabic OCR is still commonly scored by character error rate (CER) computed after diacritics are removed. We fine-tune the vision-language OCR model dots.mocr with LoRA on synthetic and human-verified classical pages and evaluate it on 124 human-verified test pages, 38 human-verified unvowelled scans and further unverified scans. We report three findings. First, letters-only CER misses most of what changes. On the test split, mean CER is unchanged (0.051 vs. 0.051), while the diacritic error rate (DER) falls from 0.204 to 0.177 and invented marks per 1,000 letters fall from 36.9 to 21.6. Second, gains are large on typeset print (DER 0.053 -> 0.008) and on several scanned sets, but absent on the only human-verified low-resolution scan book. Third, replacing recognised Qur'anic quotations with canonical text, a common post-processing step, changes the conclusions: on the test split the DER improvement becomes non-significant, and on the low-resolution book a null result becomes a significant regression. We also disclose a release gate revised after a failed evaluation, bootstrap-seed sensitivity and run-to-run variance. Research materials and reproducibility This preprint investigates the limitations of character error rate (CER) as a standalone evaluation metric for classical Arabic OCR, with particular attention to diacritic fidelity and the effects of post-processing on model comparisons. The study includes experiments comparing a baseline and fine-tuned dots.mocr model, with separate analyses of typeset and scanned Arabic pages. The fine-tuned model is publicly available through the Pandan Labs Hugging Face repository. An accompanying research dossier documents the experimental methodology, evaluation results, limitations, and reproducibility information. Research dossier: https://pandanlabs.com/research/arabic-ocr-diacritics/ Fine-tuned model: https://huggingface.co/pandanlabs/dots-mocr-arabic-classical Research transparency: The manuscript contains a detailed disclosure of generative AI assistance, including its use in code development, experiment execution, analysis, and manuscript preparation. The author retains responsibility for the research decisions, verification, and reported findings. Publication status: Independent research preprint; not peer reviewed.