Large corpora and large language models: a replicable method for automating grammatical annotation

Abstract Much linguistic research relies on annotated datasets of features extracted from text corpora, but the rapid quantitative growth of these corpora has created practical difficulties for linguists to manually clean and annotate large data samples. In this paper, we present a method that leverages large language models for assisting the linguist in grammatical annotation through prompt engineering, training, and evaluation. We apply this methodological pipeline to the case study of formal variation in the English evaluative verb construction “ consider X (as) (to be) Y”, based on the large language model Claude 3.5 Sonnet and data from Davies’s NOW and Sketch Engine’s EnTenTen21 corpora. Overall, we reach a model accuracy of over 90 % on our held-out test samples with only a small amount of training data, validating the method for the annotation of very large quantities of tokens of the construction in the future. We discuss the generalizability of our results for a wider range of case studies of grammatical constructions and grammatical variation and change, underlining the value of AI copilots as tools for future linguistic research, notwithstanding some important caveats.