Can ESL Instructors Spot Machine Translation? Evidence from Arabic-English Classroom

This study investigates the ability of ESL instructors to differentiate between AI machine-generated and student-generated translations and assesses their confidence in doing so. Twenty instructors evaluated 44 translations (22 student-generated, 22 machine-generated), classifying each as either machine- or student-produced. In total, 434 valid responses were analyzed using a Generalized Linear Mixed Model (GLMM) in R. The responses were coded as 0 (incorrect/unsure) and 1 (correct) to determine whether instructors’ correct identification of machine-generated translations was statistically significant. The results revealed a low identification rate, with instructors having only a 28% probability of correctly distinguishing machine translations, which was significantly below the 50% expected by random chance. Despite this low success rate, 90% of instructors expressed confidence in their ability to detect machine translation. The findings suggest a gap between instructors’ perceived and actual abilities, indicating that reliance on instructors’ judgments may no longer be sufficient for detecting machine translation use. Moreover, instructors may need to prioritize in-class assessments, especially when the focus is on tapping into students’ raw abilities, while take-home assignments should incorporate active post-editing to foster critical engagement with machine translation. These adjustments are crucial for maintaining academic integrity, equipping students with the skills needed to navigate both human and machine-generated translations, and preparing them for the evolving translation marketplace.