Whose Meaning Is It Anyway? Measuring Semantic Similarity in AI-Revised L2 Writing Across Genres and Models

Large language models (LLMs) are widely used to revise second language (L2) writing, raising questions about whether revision preserves writers’ meaning, stance, and voice. This study examines segment-level semantic similarity between original and AI-revised texts across three genres and four LLMs. The corpus comprised 72 texts by undergraduate English majors: 24 argumentative essays, 24 personal statements, and 24 critiques. Each text was revised by ChatGPT, Gemini, DeepSeek, and Doubao using a standardised prompt. Original and revised texts were manually segmented and aligned into semantically corresponding units. Similarity was measured using all-MiniLM-L6-v2 sentence embeddings and cosine similarity. One-way and two-way ANOVAs and Tukey HSD tests examined genre and model differences; a stratified sample of high- and low-similarity pairs was qualitatively coded. Overall, similarity scores were relatively high, but significant genre and model effects and a genre-by-model interaction emerged. Mean similarity was highest for critiques, followed by personal statements and argumentative essays. Doubao yielded the highest mean similarity, followed by ChatGPT, DeepSeek, and Gemini. Qualitative analysis indicated that high-similarity revisions mainly involved verbatim retention, surface correction, and meaning-preserving paraphrase, whereas lower-similarity revisions featured rhetorical reframing, shifts in voice or register, expansion, condensation, and deletion. These findings suggest that semantic preservation depends on both genre and model, and that embedding-based metrics require qualitative interpretation. They support genre-sensitive, model-aware, and critically guided use of LLMs in L2 writing instruction.

Authors

Institutions

Publication Details

Published
2026-11-17
DOI
https://doi.org/10.33774/coe-2026-s6lvc
Primary Topic
Discourse Analysis in Language Studies
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Whose Meaning Is It Anyway? Measuring Semantic Similarity in AI-Revised L2 Writing Across Genres and Models

Yang Liu, Xinyang Feng
4 links
Discourse Analysis in Language Studies
preprint

Whose Meaning Is It Anyway? Measuring Semantic Similarity in AI-Revised L2 Writing Across Genres and Models

Yang Liu, Xinyang Feng
preprint en

Abstract

Large language models (LLMs) are widely used to revise second language (L2) writing, raising questions about whether revision preserves writers’ meaning, stance, and voice. This study examines segment-level semantic similarity between original and AI-revised texts across three genres and four LLMs. The corpus comprised 72 texts by undergraduate English majors: 24 argumentative essays, 24 personal statements, and 24 critiques. Each text was revised by ChatGPT, Gemini, DeepSeek, and Doubao using a standardised prompt. Original and revised texts were manually segmented and aligned into semantically corresponding units. Similarity was measured using all-MiniLM-L6-v2 sentence embeddings and cosine similarity. One-way and two-way ANOVAs and Tukey HSD tests examined genre and model differences; a stratified sample of high- and low-similarity pairs was qualitatively coded. Overall, similarity scores were relatively high, but significant genre and model effects and a genre-by-model interaction emerged. Mean similarity was highest for critiques, followed by personal statements and argumentative essays. Doubao yielded the highest mean similarity, followed by ChatGPT, DeepSeek, and Gemini. Qualitative analysis indicated that high-similarity revisions mainly involved verbatim retention, surface correction, and meaning-preserving paraphrase, whereas lower-similarity revisions featured rhetorical reframing, shifts in voice or register, expansion, condensation, and deletion. These findings suggest that semantic preservation depends on both genre and model, and that embedding-based metrics require qualitative interpretation. They support genre-sensitive, model-aware, and critically guided use of LLMs in L2 writing instruction.

University of Cambridge (GB), Ocean University of China (CN)
Discourse Analysis in Language Studies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon