An inquiry into Grammarly’s precision and recall: A comparative study with human annotators

The present study aimed to investigate Grammarly’s precision and recall using a corpus of 53 samples of argumentative writing. The analysis of Grammarly’s error flaggings and corrections led to the development of a coding scheme in which various error types and error categories were identified. Next, these flaggings and corrections were tallied, and comparatively, juxtaposed with a coding scheme developed by two human annotators. The analysis of Grammarly’s performance regarding precision (i.e., error detection and correction) shows that the system was highly accurate in providing corrections for the flagged errors. However, further analysis showed that Grammarly’s recall rate was 51 %, which means that the tool failed to identify almost half of the sentence-level issues identified by the human annotators. Overall, the results suggest that, while Grammarly is highly precise in correcting the errors it flags, this system cannot effectively identify many errors second language (L2) learners may make in their writing. These results suggest that Grammarly has the potential to serve as a supplementary tool and an additional source of corrective feedback in the L2 writing classroom. However, its low recall rate should be brought to their attention, as this could have detrimental effects on their writing quality.