Training diagnostic competencies (TRACE)

The TrACE project investigated how reliably teachers evaluate English essays written by up-per-secondary students and how this accuracy can be strengthened through training and au-tomated support. Using several thousand authentic argumentative essays from the Swiss MEWS project, TrACE first established a robust human rating framework focusing on three central dimensions of writing quality: language, structure, and content. These analyses showed that the dimensions are clearly distinguishable, enabling targeted feedback to learn-ers. They also revealed how student writing develops over time: while text structure improved most noticeably over the school year, gains in content were moderate and progress in lan-guage quality remained comparatively slow, with vocabulary emerging as a particular challenge. Building on these human ratings, the project developed automated essay scoring models that combine interpretable linguistic features with modern language-processing methods. These models successfully mirrored human judgments and offered transparent insights into why a text was scored in a particular way. Because they provide concrete, pedagogically meaningful information, they hold promise for formative classroom assessment as well as for future appli-cations in teacher education. Complementary analyses further showed that specific linguistic profiles can differentiate proficiency levels across writing tasks, underlining the potential of automated tools to support consistent, criterion-based assessment. The central focus of the project was the question of what explains the accuracy of teachers' judgments. By comparing thousands of assessments by teacher training students and teach-ers with human and automated machine reference values, it became clear that the quality of judgment varies greatly and is influenced by several factors. Experienced teachers evaluated the texts more critically and thus less accurately than teacher training students, while more careful reading and comparing several texts increased accuracy. Two pre-registered experi-ments investigated whether simple aids can improve the quality of judgment. Highlighting spelling mistakes increased accuracy in this area without bias, while highlighting argumenta-tive structures did not improve analytical accuracy but led to more precise overall assess-ments and reduced a recurring halo effect. In addition, the project developed a novel calcula-tion model that allows three previously independent components of judgment accuracy to be estimated simultaneously. This model is likely to be used frequently in related research.