Transforming clinical assessments : rater reliability in video-based evaluation of upper extremity after stroke

Introduction: Compensatory movements often occur during motor recovery following a stroke and lead to negative longterm consequences. Deep learning methods have demonstrated promising potential for evaluating movement quality. However, to create a valid ground truth based on therapist’s assessment of compensatory movements, reliability of these ratings must be ensured. This study aimed to evaluate the reliability of therapists’ visual assessments of compensatory movements using 2Dvideo analysis. Material and Methods: Twenty-two occupational and physical therapists evaluated video recordings showing a standardized drinking task performed by seven patients after stroke using an online system (scale from 0 = no compensation to 100 = max. compensation). We analyzed data using a generalized-linear mixed effects model with zeroinflated beta regression. Results: Our cohort demonstrated moderate to excellent reliability for the phases of reaching, drinking, and returning, and poor reliability for smoothness and interjoint coordination. Wide Credible Intervals for the ICCs across all measured domains demonstrate high degree of uncertainty. Discussion: We propose three approaches to overcome the issue of difficulties in conducting video analysis: increasing the sample size of raters (1), providing therapists with training for calibration on key metrics of compensatory movements and the labeling system (2), and integrating additional kinematic motion analysis technologies (3).