Toward a Standardized Collation Sequence for Arabic Words with Special Characters: A Comparative Study of Digital Sorting Systems

Arabic is a language characterized by a large number of special characters, such as accents and symbols, in its script beyond alphabetic letters. While Arabic adheres to a fixed alphabetical order, the arrangement of words containing these special characters remains a matter of controversy. We examine the level of sophistication of the Arabic alphabetical sorting systems implemented in Microsoft Office Word and Excel documents, as well as in Python, which utilizes UTF-8 encoding. A list of Arabic words was carefully selected for evaluation purposes. Each group of words in the list shared almost the same consonantal root but with varying characters and diacritics. Extraction and comparison of the sorted outputs from the three programs revealed marked sorting differences in the three sorted lists, with discrepancies as significant as 58% observed across the tested conditions. To address this issue, we propose a secondary alphabetical order for special characters that extends beyond the primary order of Arabic letters. The order is based on some linguistic features of the special characters, such as the root of the word and the phonological salience of the character. Software developers working with Arabic script in digital applications are advised to incorporate the recommendations of this study and to make adjustments to the alphabetical collation algorithms implemented within their programs.