Using deep learning to detect upper limb compensation in individuals post-stroke using consumer-grade webcams—A feasibility study

As societies age, the number of individuals experiencing stroke increases, necessitating more effective rehabilitation strategies. Over half of stroke survivors suffer from upper limb impairments, making assessments of sensory-motor function crucial for both improving interventions and tracking progress. Ideally, such assessments could also be performed at home without requiring a therapist's presence. Advances in computer vision and human pose estimation allow for human movement analysis using consumer-grade cameras. This study investigates whether a single webcam, combined with human pose estimation and deep learning algorithms, can automatically detect compensatory movements in persons with stroke performing a drinking task. Twenty participants with stroke with mild to moderate upper limb impairment were recruited. Each participant performed multiple repetitions of the drinking task while being recorded by multiple cameras and an optical motion capture system (OMC) for kinematic ground truth. The videos were labeled by therapists to indicate the presence or absence of compensatory movements. Human poses were extracted from the videos using MediaPipe, and deep learning models were trained to predict these compensatory movements based on MediaPipe keypoints. Several factors affecting compensation detection accuracy were evaluated. Models trained on raw MediaPipe keypoints for inter-person compensation detection failed to generalize, achieving accuracy around 50%. Using custom features instead of raw keypoints improved the accuracy to 70%. In contrast, intraperson classification achieved high accuracy, typically exceeding 90%. Using OMC data significantly improved classification accuracy compared to using MediaPipe keypoints. Camera angle had an effect on accuracy, and convolutional neural networks outperformed long short-term memory networks. Generalizing models remain limited by (1) the measurement uncertainty of human pose estimation and (2) insufficient data representing the full spectrum of compensatory strategies (3) accurate compensation labels. The results demonstrate that deep learning approaches can differentiate between compensatory and non-compensatory movements when movement representations are sufficiently accurate. Future work should improve pose estimation and expand labeled datasets to better reflect the stroke population. While general models are limited in accuracy, personalized models using consumer cameras can support home-based rehabilitation. This digitalized assessment approach has the potential to quantify recovery progress throughout the continuum of care.