Learning distribution-calibrated and prompt-robust quality representations for cross-prompt automated essay scoring inEnglish writing assessment

Abstract In English language teaching and writing assessment, cross-prompt automated essay scoring (AES) requires models trained on existingsource prompts to generalize to unseen prompts without relying on target-prompt responses. A central challenge is that prompts differ not only in rubric ranges but also in score distributions and optimization characteristics, so numerical score normalization alone may not provide transferable supervision. This study proposes DCR-AES, a source-only framework that combines prompt-relative empirical-distribution calibration, separation-filtered cross-prompt relative-quality learning, and prompt-level distributionally robust optimization. Human scores are mapped to mid-rank empirical cumulative distribution function (eCDF) positions estimated exclusively from source-prompt training data. These positions are interpreted as prompt-relative ordinal coordinates rather than absolute cross-prompt quality equivalents and are used to construct ranking constraints only when the calibrated separation is sufficiently large. Prompt-level robust optimization further adapts source-domain contributions according to persistent training loss while retaining a uniform contribution from all source prompts. Under strict leave-one-prompt-out evaluation on ASAP++, the five-run analysis yields a prompt-level QWK of 0.593 ± 0.004 (95% CI [0.588, 0.598]) and an attribute-level QWK of 0.590 ± 0.004. DCR-AES improves over PLAES by 0.0179 QWK across the eight held-out prompts (Wilcoxon p = 0.0078; rank-biserial effect size = 1.00), does not differ significantly from ProTACT, and remains below SICTS and stronger resource-enhanced systems in absolute QWK. Additional analyses show that calibration gains are positively associated with source-target score-distribution shift, while learned DRO weights correlate strongly with persistent training loss but not with static sample count or score-distribution descriptors. The results support prompt-relative distribution information as useful auxiliary supervision while clarifying the limits of percentile alignment and loss-based prompt weighting.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-12
DOI
https://doi.org/10.1038/s41598-026-78832-4
Primary Topic
Natural Language Processing Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Learning distribution-calibrated and prompt-robust quality representations for cross-prompt automated essay scoring inEnglish writing assessment

郇怡斌
Scientific Reports
Natural Language Processing Techniques
article

Learning distribution-calibrated and prompt-robust quality representations for cross-prompt automated essay scoring inEnglish writing assessment

郇怡斌
article en

Abstract

Abstract In English language teaching and writing assessment, cross-prompt automated essay scoring (AES) requires models trained on existingsource prompts to generalize to unseen prompts without relying on target-prompt responses. A central challenge is that prompts differ not only in rubric ranges but also in score distributions and optimization characteristics, so numerical score normalization alone may not provide transferable supervision. This study proposes DCR-AES, a source-only framework that combines prompt-relative empirical-distribution calibration, separation-filtered cross-prompt relative-quality learning, and prompt-level distributionally robust optimization. Human scores are mapped to mid-rank empirical cumulative distribution function (eCDF) positions estimated exclusively from source-prompt training data. These positions are interpreted as prompt-relative ordinal coordinates rather than absolute cross-prompt quality equivalents and are used to construct ranking constraints only when the calibrated separation is sufficiently large. Prompt-level robust optimization further adapts source-domain contributions according to persistent training loss while retaining a uniform contribution from all source prompts. Under strict leave-one-prompt-out evaluation on ASAP++, the five-run analysis yields a prompt-level QWK of 0.593 ± 0.004 (95% CI [0.588, 0.598]) and an attribute-level QWK of 0.590 ± 0.004. DCR-AES improves over PLAES by 0.0179 QWK across the eight held-out prompts (Wilcoxon p = 0.0078; rank-biserial effect size = 1.00), does not differ significantly from ProTACT, and remains below SICTS and stronger resource-enhanced systems in absolute QWK. Additional analyses show that calibration gains are positively associated with source-target score-distribution shift, while learned DRO weights correlate strongly with persistent training loss but not with static sample count or score-distribution descriptors. The results support prompt-relative distribution information as useful auxiliary supervision while clarifying the limits of percentile alignment and loss-based prompt weighting.

Scientific Reports
Zhoukou Normal University (CN)
Quality education
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon