GenAI Models as Keyword Rankers: A Learner-centred Case Study for L2 Spanish

Frequency-based word lists form an important part of general-purpose vocabulary learning courses aimed at beginner and (lower-)intermediate learners of a foreign/second language (L2). For advanced learners and/or specific purposes, however, relying exclusively on these general word lists will be unlikely to lead to an adequate selection of vocabulary. As research in this latter area remains scarce (especially for languages other than English), the present study aims to fill (part of) the gap by investigating the use of Generative Artificial Intelligence (GenAI) models to automatically rank vocabulary items based on how typical they are of a given topic, focusing on Spanish as the target language. I compile a dataset containing four domain-specific subsets of 200 vocabulary items (for the topics economics, health, law, and migration) and analyse how well GenAI-based rankings of these vocabulary items (using zero-shot prompting) correlate with gold standard human rankings (provided by L2 learners). As the evaluation baseline, I use the rankings obtained by means of the Kullback-Leibler divergence (i.e., a statistical keyness measure based on word frequencies). With a top average Spearman’s ρ and Kendall’s weighted τ of 0.73, this first-of-its-kind study demonstrates that the tested GenAI models (Gemma, Llama, and Mistral) outperform the baseline by a large margin, showing great potential for use in the real-life creation of domain-specific vocabulary lists for L2 learning purposes.