Prompt Sensitivity in AI Writing: A Methodological Caution for Corpus‐Based Analysis of AI‐Generated Language

ABSTRACT Recent corpus‐based studies suggest generative AI systems such as ChatGPT can approximate human patterns of interpersonal metadiscourse in argumentative writing. These conclusions are typically based on AI‐generated texts produced using a single prompt, despite AI output being highly sensitive to even minor changes in prompt specification. This study examines prompt specificity as an elicitation variable in the production of metadiscourse. Using Hyland's interpersonal metadiscourse framework, we first compared university students’ argumentative essays from the LOCNESS corpus with GPT‐generated essays produced in response to a general essay (baseline) prompt specifying role, genre, topic, length and persuasive purpose, but providing no explicit guidance regarding metadiscourse use. We then generated a second GPT corpus using a metadiscourse‐inducing prompt that explicitly requested interactive, interactional, and engagement features that were found to be underrepresented in the initial GPT corpus relative to the LOCNESS corpus. The baseline prompt produced lower frequencies of transitions, hedges, self‐mention, attitude markers and engagement resources than human writing, while the metadiscourse‐inducing prompt substantially increased several target features over the baseline and human conditions. Some categories converged with baseline frequencies, but even where convergence occurred, these were often realised in more formulaic ways. The findings show prompt specificity significantly shapes both the distribution and rhetorical realisation of metadiscourse in AI‐generated writing. More broadly, the results highlight the importance of treating prompt specification as a key methodological variable when interpreting linguistic patterns in AI‐generated texts.