Benchmarking Arabic Transformer‐Based Language Model Embeddings for Document Clustering: A Comparative Study

ABSTRACT Recent advances in transformer‐based language models have substantially improved semantic text representations for Arabic natural language processing. However, their effectiveness for unsupervised document clustering has not been comprehensively investigated across different clustering algorithms, preprocessing strategies and document representations. This study presents a comparative evaluation of six transformer‐based embedding models, namely AraGPT2, arabertv0.2, CAMeLBERT, AraELECTRA, AraBERTv2, and Multilingual‐MiniLM, together with a TF‐IDF lexical baseline, for Arabic document clustering. The evaluation was conducted on five Arabic datasets spanning both general‐domain and domain‐specific corpora using four clustering algorithms: K ‐Means++, Spherical K ‐Means, Hierarchical Agglomerative Clustering (HAC) and Spectral Co‐Clustering. To investigate the influence of text representation, six preprocessing configurations were considered by varying stemming and document‐length reduction strategies. Clustering performance was assessed using Normalised Mutual Information (NMI), Accuracy, Purity, Adjusted Rand Index (ARI), Macro F 1‐score, V‐Measure and Entropy. Furthermore, the experimental results were statistically validated using the Friedman test, average‐rank analysis, and Holm‐corrected Wilcoxon signed‐rank post hoc comparisons. The experimental results demonstrate that preserving the original lexical and morphological structure of Arabic text generally yields superior clustering performance for contextual embedding models compared with light stemming. The influence of document length was found to be dataset‐ and representation‐dependent, with full‐document representations achieving the strongest overall performance, while shorter representations frequently remained competitive. TF‐IDF provided a strong lexical baseline, particularly for general‐domain datasets. Among the evaluated embedding models, AraBERTv2 achieved the best overall average rank, closely followed by arabertv0.2, and the post hoc statistical analysis revealed no significant difference between the two models. CAMeLBERT consistently exhibited stable and competitive performance across all datasets, whereas AraELECTRA produced the weakest clustering results. From the clustering perspective, Spherical K ‐Means demonstrated the most consistent overall performance, while HAC achieved the highest clustering quality in several of the best‐performing embedding configurations. Overall, this study provides a comprehensive benchmark for transformer‐based Arabic document clustering and offers practical guidance for selecting embedding models, preprocessing strategies, document representations, and clustering algorithms for unsupervised Arabic text analysis.