Uncovering the Differences between LLM-generated and Human-written Answers to an Ideational Creativity Test

In recent years, Large Language Models proved themselves worthy competitors of humans in various creative domains. Studies have shown them to be on a par with humans or even better, when tested for several types of creativity. However, the nature of their creative process is fundamentally different from humans, which leads to differences in their creative output, which are not yet fully understood. In this study, we investigate the degree and the nature of the differences between Large Language Models and human answers from a dataset composed of responses to tan ideational creativity test previously proposed in in the literature. We automatically analyzed the dataset as follows. We first extracted socio-linguistic features and computed the Linguistic Style Matching score from the dataset, using the LIWC tool. Then we performed style embeddings clustering to see how the human and LLM answers to the creativity test group together. Moreover, we computed and compared several types of readability indexes. Finally, we performed sentiment analysis and topic modeling on the dataset. The results showed a variety of dissimilarities between human and LLM answers to the creativity test, from significative differences in stop words, part of speech, and tense usage, to tone, emotions, and readability. Moreover, LLMs tend to generate more general and impersonal creative responses than humans, who rely more on their own experience and expectations.