Comparative analysis of AI-generated and original abstracts of academic articles on philology

Generative artificial intelligence systems have a significant impact on tasks related to natural language processing: machine translation, sentiment analysis, text generation, and summarisation, etc. The aim of the presented work was to determine the features of automatically generated academic texts in comparison with texts created by authors, and to evaluate the capabilities of different methods in relation to the task of their classification. The paper analyses two types of abstracts: collected from academic journals on computational linguistics and Germanic studies and generated from the titles of the corresponding articles using ChatGPT-4o mini. The total amount of data was 60 items. The choice of article topics is due to the fact that the texts belong to the same subject area but differ in their structure. The first group which contains original texts on computational linguistics, is similar to the abstracts of academic articles on computer science, and contains a large amount of English terminology. The second group contains texts on Germanic studies and is more descriptive-narrative in their nature. We analyzed the differences between the two types of abstracts and classified them into two categories with the help of experts, three detector systems to determine the involvement of artificial intelligence in the creation of texts (Smodin, ZeroGPT and GPTZero), as well as the ChatGPT system itself. The analysis showed that the generated texts are characterized by a clear formal structure and adherence to the rules of academic text construction in accordance with IMRAD (Introduction, Methods, Results and Discussion). They are superficial in content and they do not always follow the scientific style; there are repetitions of constructions and paraphrasing of article titles, which is not found in the abstracts written by the authors without artificial intelligence. Automatically generated abstracts need not only further editing (because in some cases lexical and syntactic coherence is broken and ambiguity is present), but also verification of the facts and terms mentioned. Among the detector systems, the highest scores in Precision, Accuracy and F1-score are achieved by Smodin tools, while the best results in Recall are achieved by ZeroGPT. The lowest results in abstract evaluation when compared with other tools were achieved by the ChatGPT system itself. Expert-assisted classification showed the highest results in the case of Germanic abstracts. The results may be useful for researchers when working with academic texts on linguistics as well as for further finetuning of neural network models.