Feature comparison between human-written and LLM-generated documents – a case study

Abstract In this paper, we explored features of human-written (HW) and large language model-generated (LLMG) texts, specifically on a dataset of Polish student reports. Expanding upon the previous research with email datasets, we constructed a dataset containing 2,400 pairs of original student-written reports and their LLMG counterparts. Using a feature-based detection approach, we analyzed 220 distinct features, including perplexity, burstiness, error frequency, stylometric elements, and sentence-level characteristics. Our findings reveal that LLMG texts demonstrate lower perplexity and burstiness values, fewer errors, greater lexical diversity, and more consistent sentence lengths compared to HW texts. Among stylometric features, relative pronouns, stop words, and content word diversity emerged as notably discriminative. The preliminary classification results, achieved with the proposed detection model, confirm the viability of feature-based methods for reliably discriminating between HW and LLMG documents.