Assessing the Quality of Human and GPT-Generated Paraphrases in English: An NLP-Based Approach

This study examines the quality of paraphrases generated by humans and by GPT-4o through NLP-based metrics. Using 500 sentence pairs from the Microsoft Research Paraphrase Corpus as human-produced samples, an equal number of paraphrases were generated by GPT-4o under standardized prompts. Three evaluation metrics were applied: syntactic diversity (dependency distance), semantic similarity (cosine similarity), and lexical diversity (Jaccard distance). Statistical analyses revealed that GPT-4o significantly outperformed human writers in semantic similarity and syntactic diversity, while showing a marginal advantage in lexical diversity. Correlation analysis indicated that GPT-4o tends to preserve semantic meaning with moderate syntactic and lexical shifts, whereas human paraphrases often sacrificed semantic congruence for greater variation. Furthermore, the difference between the frequency of paraphrasing strategies between Human and GPT data set was also investigated arguing possible relationship between strategy preference and paraphrasing scores. A qualitative analysis further examined the effect of the temperature parameter on GPT-4o’s outputs. Higher temperature values led to longer, more elaborate paraphrases with greater stylistic variation, indicating that temperature functions as a control mechanism for output complexity. These findings have implications on the field of human-to-machine research as well as educational, evaluative, and generative applications of AIs where paraphrasing is essential.