Out-Heroding Herod? — Author-trained GPTs and Original Works in the Perspective of Quantitative Linguistics

Goals: The paper compares texts created by GPT models trained on the works of prominent Czech authors and the pieces of literature they actually wrote. The goal is to find out (1) whether there are any differences between the two; and if so, (2) in what sphere of language these differences are the most prominent. Methods: The authors used for building GPTs are Karel Čapek, Jaroslav Hašek, Franz Kafka, and Vladislav Vančura. The corpus contains 40 1,000-word text samples per each, 20 of them produced by the respective GPT and 20 taken from the original works. Two investigations are carried out – the first includes calculating 30 morphological, syntactic, and lexical markers for each text; the second is based on most-frequent-element analyses. The results of the first set are tested on statistical significance via Mann–Whitney U Test. Results: The chatbots do not reflect colloquiality of style and conversation interaction very well, and tend to make texts more narrative. The best results are obtained for Karel Čapek, the worst for Franz Kafka. The stylometric analyses almost always distinguish the AI- and human-generated pieces of language. Conclusions: The texts produced by the author-trained GPTs are still very well distinguishable from those produced by real writers.