Scaffolded large language model feedback builds durable translation competence whereas unrestricted use fosters dependency

Abstract Large language models (LLMs) are rapidly entering translator education, yet it remains unclear whether they build students' own translation competence or merely improve the products students submit. We combined a validation study on public benchmark data with a 16-week quasi-experiment to test a scaffolded human--LLM pedagogy, the Draft--Diagnose--Revise--Reflect (D2R) cycle, for English--Chinese translation. In Study 1, a rubric-guided error-annotation prompt increased the agreement between LLM feedback and expert Multidimensional Quality Metrics judgements on 2,500 WMT segments for all four models tested; with this prompt GPT-4o reached a segment-level Kendall's $\tau$ of 0.30, on a par with the strongest neural metric, and the highest error-detection accuracy. In Study 2, 179 English majors received conventional instruction, unrestricted LLM access or the guardrailed D2R cycle. Unrestricted access maximised LLM-assisted performance but yielded no advantage in unassisted translation over conventional instruction at a delayed test 12 weeks after the course (Hedges' $g$ = $-$0.08) and produced the largest AI-dependency gap. Scaffolded use produced durable gains in unassisted translation ($g$ = 0.70), a 13\% reduction in error rate and higher metacognitive regulation. Process logs indicated that the benefit was transmitted mainly through lower verbatim adoption of LLM output. Pedagogical design, not LLM access per se, determines whether LLMs support or supplant translation learning.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-16
DOI
https://doi.org/10.21203/rs.3.rs-11512296/v1
Primary Topic
Translation Studies and Practices
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Scaffolded large language model feedback builds durable translation competence whereas unrestricted use fosters dependency

Guangyin Jin, Xixi Zhang
Research Square
Translation Studies and Practices
preprint

Scaffolded large language model feedback builds durable translation competence whereas unrestricted use fosters dependency

Guangyin Jin, Xixi Zhang
preprint en

Abstract

Abstract Large language models (LLMs) are rapidly entering translator education, yet it remains unclear whether they build students' own translation competence or merely improve the products students submit. We combined a validation study on public benchmark data with a 16-week quasi-experiment to test a scaffolded human--LLM pedagogy, the Draft--Diagnose--Revise--Reflect (D2R) cycle, for English--Chinese translation. In Study 1, a rubric-guided error-annotation prompt increased the agreement between LLM feedback and expert Multidimensional Quality Metrics judgements on 2,500 WMT segments for all four models tested; with this prompt GPT-4o reached a segment-level Kendall's $\tau$ of 0.30, on a par with the strongest neural metric, and the highest error-detection accuracy. In Study 2, 179 English majors received conventional instruction, unrestricted LLM access or the guardrailed D2R cycle. Unrestricted access maximised LLM-assisted performance but yielded no advantage in unassisted translation over conventional instruction at a delayed test 12 weeks after the course (Hedges' $g$ = $-$0.08) and produced the largest AI-dependency gap. Scaffolded use produced durable gains in unassisted translation ($g$ = 0.70), a 13\% reduction in error rate and higher metacognitive regulation. Process logs indicated that the benefit was transmitted mainly through lower verbatim adoption of LLM output. Pedagogical design, not LLM access per se, determines whether LLMs support or supplant translation learning.

Research Square
Nanjing University of Science and Technology (CN), Wuhan College of Arts and Science (CN)
Translation Studies and Practices
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon