AI text detectors and the misclassification of slightly polished Arabic text

Abstract Many artificial intelligence (AI) detection models have been developed to counter the presence of articles generated by AI. However, AI-based slight polishing to an authentic human-written article can shift the borderline decision of detection models, leading them to consider it as an AI-generated. This misclassification risks falsely accusing authors of plagiarism and harming the credibility of the detectors. In English, some efforts were made to meet this challenge, but not in Arabic. To address this gap, we generated two datasets. The first dataset contains 800 Arabic articles, half AI-generated and half human-written articles. We used it to evaluate 14 Large Language Models (LLMs) and commercial detectors to assess their ability to distinguish between human-written and AI-generated articles. The best eight models were chosen to act as detectors for our primary concern, which is whether they would misclassify slightly polished human-written text as AI-generated. The second dataset, Ar-APT, contains 400 Arabic human-written articles polished by 10 LLMs using 4 polishing settings, totaling 16400 samples. We use it to evaluate the 8 nominated models. The results reveal that all AI detectors incorrectly attribute a significant number of articles to AI. The best performing LLM on human-written articles, Claude-4 Sonnet, achieved 83.51% human-only accuracy, its performance decreased to 57.63% for articles slightly polished by LLaMA-3 70B. Whereas the best performing commercial model, Originality.AI, achieves 92% human-only accuracy, dropped to 12% for articles slightly polished by Mistral.