Language Modeling for Sentence Level Assessment: A Case Study of First-Year English Composition

Recent advancement in neural language models has witnessed many applications in writing instruction and assessment, including automated writing assessment in secondary and post-secondary education. Using language models for student writing assessment brings promises of increased accuracy, speed, and objectivity in evaluating student writing, benefiting writing students and teachers. Based on our case study using sentence-level prediction in Writing Across the Curriculum (WAC) assessment of College Writing 1 and 2 courses at the Florida Atlantic University (FAU), we argue in this paper that automatic writing assessment should be implemented at the sentence level. We collected end-of-semester argumentative essays, which were then systematically evaluated at the sentence- and the whole-document level. The study shows how neural language models can achieve a certain level of accuracy, but scoring inconsistency is a significant challenge to the learning model. We propose solutions to address such inconsistency, and also report important findings from the case study.