Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study

The performance of conversational Large Language Models (LLMs) in general, and of ChatGPT in particular, is currently being evaluated on many different tasks, from logical reasoning or math to answering questions on a myriad of topics. Instead, much less attention is being devoted to the study of the linguistic features of the texts generated by these LLMs. This is surprising since LLMs are models for language, and understanding how they use the language is important. Indeed, conversational LLMs are poised to have a significant impact on the evolution of languages as they may eventually dominate the creation of new text. This means that for example, if conversational LLMs do not use a word it may become less and less frequent and eventually stop being used altogether. Therefore, evaluating the linguistic features of the text they produce and how those depend on the model parameters is the first step toward understanding the potential impact of conversational LLMs on the evolution of languages. In this article, we consider the evaluation of the lexical diversity of the text generated by LLMs in English and how it depends on the model parameters. A methodology is presented and used to conduct a comprehensive evaluation of lexical diversity using ChatGPT as a case study. The results show how lexical diversity depends on the version of ChatGPT and some of its parameters, such as the presence penalty, or the role assigned to the model. The dataset and tools used in our analysis are released under open licenses with the goal of drawing much-needed attention to the evaluation of the linguistic features of LLM-generated text.