Examining reliability in translation quality assessment

Abstract This study examines the differences between paper- and computer-based translation quality assessment, focusing on score reliability, variability, scoring speed, and raters’ preferences. Utilizing a within-subjects design, 27 raters assessed 29 translations presented in both handwritten and word-processed formats, employing a holistic scoring method. The findings reveal comparable translation quality ratings across both modes, with paper-based scoring showing greater inter-rater disagreement and being affected by handwriting legibility. Paper-based scoring was generally faster, though computer-based scoring demonstrated less variability in inter-rater reliability. Raters exhibited a preference for paper-based scoring due to its perceived faster speed, flexibility in annotating, and eye-friendliness. The study highlights the importance of comprehensive rater training and calibration to mitigate biases and non-uniform severity, as well as the adoption of detailed scoring rubrics to ensure consistent assessment across modes. The article offers insights on refining computer-based scoring systems, including enhancements in annotation functionality and ergonomic considerations.