Evaluating the accuracy and reliability of AI content detectors in academic contexts

The rapid adoption of generative AI (GenAI) in higher education has intensified concerns about academic integrity, particularly for institutions serving English as a Foreign Language (EFL) learners. AI content detectors such as Turnitin and Originality are now widely used to identify potential misuse of GenAI in student writing, yet their accuracy, consistency, and fairness remain to be proven. This study evaluates the reliability of these two commercial detectors using a balanced dataset of 192 texts, composed of authentic EFL student writing produced before the widespread availability of GenAI, professional human-authored texts, AI-generated outputs, and hybrid compositions. Detector outputs were categorized into Human, Hybrid, or AI using established thresholding ranges, and performance was assessed using standard classification metrics alongside statistical tests of significance. Results show that Originality outperformed Turnitin in overall accuracy (0.69 vs. 0.61) and macro-average recall (0.60 vs. 0.51). However, both detectors performed poorly on Hybrid texts, an increasingly common form of student writing, indicating substantial difficulty distinguishing mixed authorship. Performance declined significantly as a result of increased text length and genre variation, with both detectors achieving noticeably lower accuracy on scientific writing than on humanities texts. Originality also showed a borderline trend toward higher accuracy on professionally written texts compared to EFL student writing, suggesting potential fairness concerns. Taken together, the findings demonstrate that while AI detectors may serve as supplementary tools—serve only as indicators that prompt further inquiry, their limitations make them unsuitable as the sole basis for decisions regarding academic misconduct. For EFL-focused institutions in particular, over-reliance on machine-based identification risks misclassifying legitimate student work. The study highlights the need for informed human judgment, clearer institutional policies on acceptable AI use, and continued development of equitable detection methods that account for linguistic diversity.