Concordancing with AI: Applications of word and sentence embeddings

• Word embeddings facilitate concordancing through deeper semantic query capabilities. • Sentence embeddings help to group and order complex concordance patterns. • Word and sentence embeddings facilitate language variation. • Integrating corpus methods with AI models leads to novel research findings. • Embedding-based concordancing supports a shift to semantic-based analysis. Concordancing is a central method in corpus research. It also plays an important role in the data-driven learning (DDL) classroom, where learners use Key-Word-In-Context (KWIC) analysis to discover and implicitly learn lexical and grammatical patterns in a specific language domain. Effective concordancing requires users to craft precise single- or multi-word queries that capture the language features of interest, but these queries can quickly become complex and error-prone. The method also relies on users selecting, ordering, and grouping results in order to interpret them in a meaningful way, which can also be a significant challenge. This paper proposes using word and sentence embedding models from the field of AI to facilitate both the querying of corpora and the interpretation of concordance results. First, the paper explains that embeddings are numerical representations of language that capture semantic and contextual information in a high-dimensional vector space. Next, the paper reports on three experiments using pre-trained embedding models (BERT, word2vec) looking at synonym searches, concordance grouping and ordering, and language variation analysis across two general English corpora. The results show that embeddings allow for ‘fuzzy’, nuanced, context aware searches of corpus data without the need for meticulous query crafting, and enable the grouping and ordering of results in novel, interesting, and useful ways. Concordancing with word embedding candidates – The case of “pavement”.