Toward ground truth : evaluating therapists’ reliability in video-based assessment of upper limb compensatory movements post-stroke

Background: Upper limb compensatory movement patterns, possibly leading to negative long-term consequences, often occur during motor recovery post stroke [1]. Deep learning methods have shown promise in assessing movement quality [2]. Ensuring reliability in assessing compensatory movements is crucial for developing an accurate measurement tool and valid ground truth. This study aimed to evaluate the reliability of therapists’ visual assessments in evaluating compensatory movement patterns using video analysis. Methods: 22 OTs and PTs labeled video recordings showing a standardized drinking task performed by 7 patients after stroke using an online system (scale from 0 = no compensation to 100 = max. compensation). The post-stroke patients had mild to moderate upper limb motor impairments. We analyzed data using a generalized-linear mixed effects model with zero-inflated beta regression [3]. Results: Our cohort demonstrated good interrater reliability for the reaching (ICC: 0.76, CI: 0.44-1), drinking (ICC: 0.88, CI: 0.60-1) and returning (ICC: 0.75, CI: 0.43-1) phases. The intrarater reliability was good to excellent for the forward transporting (ICC: 0.84, CI: 0.20-1), returning (ICC: 0.76, CI: 0.21-1) and drinking (ICC: 1, CI: 0.24-1) phases. However, the global domains of smoothness and interjoint coordination showed poor interrater (ICC Smoothness: 0.49, ICC Interjoint coordination: 0.40) and intrarater reliability (ICC Smoothness: 0.13, ICC Interjoint coordination: 0.19), with CI’s ranging from 0.03-1. The wide CIs for the ICCs across all measured domains suggest that our results may not be replicable [3]. Conclusion: Based on our results we cannot recommend establishing a ground truth for the automated assessment of compensatory movement patterns based solely on therapists’ video-based assessments. These findings underscore the need for further research to determine how therapists’ ratings can be effectively integrated in the process of building a ground truth [3].