Analyzing the Effect of Linguistic Instructions on Paraphrase Generation

Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 755–766 March 3-4, 2025 ©2025 University of Tartu LibraryAnalyzing the Effect of Linguistic Instructions on Paraphrase Generation Teemu Vahtola1 Songbo Hu2 Mathias Creutz1 Ivan Vuli´c2 Anna Korhonen2 Jörg Tiedemann1 1Unversity of Helsinki, Finland 2Language Technology Lab, University of Cambridge, UK {teemu.vahtola, mathias.creutz, jorg.tiedemann}@helsinki.fi {sh2091, iv250, alk23}@cam.ac.uk Abstract Recent work has demonstrated that large language models can often generate flu- ent and linguistically correct text, adhering to given instructions. However, to what extent can they execute complex instruc- tions requiring knowledge of fundamental linguistic concepts and elaborate semantic reasoning? Our study connects an established linguis- tic theory of paraphrasing with LLM-based practice to analyze which specific types of paraphrases LLMs can accurately produce and where they still struggle. To this end, we investigate a method of analyzing para- phrases generated by LLMs prompted with a comprehensive set of systematic linguis- tic instructions. We conduct a case study using GPT-4, which has shown strong per- formance across various language genera- tion tasks, and we believe that other LLMs may face similar challenges in comparable scenarios. We examine GPT-4 from a linguistic per- spective to explore its potential contribu- tions to linguistic research regarding para- phrasing, systematically assessing how ac- curately the model generates paraphrases that adhere to specified transformation rules. Our results suggest that GPT-4 fre- quently prioritizes simple lexical or syn- tactic alternations, often disregarding the transformation guidelines if they overly complicate the primary task.