Modeling Raters’ Scoring Process in Assessment of English-Chinese Consecutive Interpreting: Insights from Qualitative Content Analysis and Social Network Analysis

In rater-mediated performance assessment, the process through which raters acquire, retrieve, and synthesize information before assigning scores is central to establishing scoring validity. While various models of scoring process have been proposed for monolingual writing and speaking assessment, the evaluation of interlingual interpreting performance has received little empirical attention. Drawing on 360 stimulated recalls (approximately 400,000 words) from 40 raters, this study explores the scoring process in the assessment of bidirectional English-Chinese consecutive interpreting, using qualitative content analysis to identify distinct scoring behaviors and social network analysis to model their interrelationships. The analyses revealed (1) 48 scoring behaviors across six dimensions, (2) dynamic progressive and recursive decision-making trajectories among scoring behaviors, and (3) a modulating effect of rater experience and direction of interpreting (i.e. whether it was from English to Chinese or Chinese to English). This study provides a systematic and fine-grained data-driven model of rater cognition. The findings can have implications for rater training, rater cognition modeling, and the validation of interpreting as assessment.