Classifying Upper Extremity Motor Function Using Deep Learning and Third-Person Video Data

Post-stroke rehabilitation of upper extremity (UE) motor function is essential. Despite the widespread use of UE rehabilitation in clinical settings, assessing the success of these treatments is challenging. Methods for evaluating UE motor function include task performance under clinical supervision, patient self-reports, and data analysis from wearable devices equipped with accelerometers and gyroscopes. In prior research, we demonstrated that machine learning and deep learning models using data from a single wrist-worn accelerometer sensor could accurately differentiate between functional and non-functional UE movements in stroke patients. To overcome the limitations of single wrist-worn accelerometer sensors – challenges in capturing the full context of functional movements – this study presents a new deep learning (DL) based framework designed to classify functional and non-functional arm movements in videos captured from a conventional camera. The system is entirely automated and comprises two DL networks. The first network performs human pose estimation, extracting 2D pose key points of the paretic arm(s) and torso on 2-second sequences of the frames. The second network then uses these 2D pose landmarks to classify the movement sequence as functional or non-functional. This system offers two key benefits for rehabilitation. First, it can automatically generate initial annotations for video frames, significantly reducing the time needed for manual labeling. Second, analyzing the effectiveness of UE rehabilitation through third-person videos allows for objective outcome measurement of UE treatments in stroke survivors' home environments.