3D human pose estimation for analyzing movement patterns during physical exercise

Marker-based motion capture is the gold standard for conducting 3D human movement analysis in rehabilitation or clinical environments. These systems comprise multiple infrared cameras and markers attached to the individuals, delivering high tracking accuracy. However, they require time-consuming marker placement, trained operators, and dedicated laboratory space, making their use expensive. In recent years, progress in machine learning and computer vision has advanced the discipline of 3D human pose estimation (HPE), which allows tracking specific human landmarks on video data, e.g., from depth or fully monocular cameras, i.e., two-dimensional. Thus, this technology holds enormous potential for human movement analysis during physical exercise outside the laboratory since it does not require a complex lab setup and eliminates the burden of expensive and time-consuming marker placement. However, as 3D HPE estimates anatomical landmarks using machine learning and does not represent physical measurements as with marker-based systems, this technology comes with limitations in tracking accuracy. For example, more complex and fast movements can negatively impact tracking accuracy, and 3D HPE based on a single camera suffers from problems such as self-occlusions or pose ambiguities. In this dissertation, we investigate using 3D HPE to analyze human movement during physical exercise. Specifically, we focus on using machine learning-based methods to analyze and quantify movement patterns obtained from 3D HPE algorithms. On this journey, we address various aspects of data-driven human motion analysis that need to be considered in further developing such systems. To this end, we start our investigation by assessing the tracking accuracy of 3D HPE algorithms. We will then look at the data-driven analysis of movement patterns to predict fatigue during resistance training. Finally, we will tackle a common problem in training machine learning models: limited availability of patient cohort data. Firstly, we analyze the motion capture accuracy of the two latest Microsoft Kinect generations, which are 3D cameras with their own associated and proprietary 3D HPE algorithms. Specifically, we investigate the tracking accuracy of the cameras for gait analysis. In our conducted study, participants walked on a treadmill at different speeds while being recorded simultaneously with the two 3D cameras and a markerbased motion capture system as reference. We then present an extrinsic camera calibration method to align the coordinate systems of the 3D cameras and the markerbased reference system. From the synchronized and calibrated three-dimensional kinematic data, we calculate gait parameters and analyze the accuracy using the ground truth. Our results show that both systems provide accurate measurements for temporal parameters, but the newer camera generation outperforms the previous one for spatial parameters. Additionally, investigate the performance of a monocular 3D HPE method by fine-tuning it to the collected gait dataset. Our results suggest that these methods could be used in rehabilitation scenarios to assess certain parameters. Secondly, we investigate using 3D HPE algorithms to predict fatigue during resistance training. Fatigue is a frequent problem in individuals when performing physical exercises, as accumulating fatigue can lead to an increased risk of injuries. Since fatigue is a complex phenomenon, we define it as subjective ratings of exertion based on a numerical scale, where subjects self-rate their current exertion levels. We conducted a study in which participants performed a fatigue protocol and rated their current fatigue level after each performed set. Participants were recorded with multiple sensor modalities, including electrocardiography, inertial measurement units (IMUs), and two 3D Kinect cameras. From there, we present fatigue prediction methods based on conventional machine learning and deep learning to predict the ratings of perceived exertion. To this end, we compare the differences in prediction accuracy when training exclusively on data from 3D HPE or IMUs. Additionally, we combine both sensor sources to train the fatigue prediction models. Our results show that combining data from sensor sources increases the prediction accuracy, indicating multi-modality benefits in this application. Lastly, when training machine learning models to analyze and quantify movements patterns, small dataset sizes are a frequent problem. Especially when involving patient cohorts, datasets might be unbalanced toward healthy controls. Therefore, we tackle this problem of small dataset size and class imbalance by proposing a scheme to generate synthetic 3D kinematic data, as provided by 3D HPE algorithms, of specific patient cohorts. We investigate the usefulness of this data augmentation method by generating synthetic samples to balance small datasets. Based on the example of a deep learning model that aims to distinguish between healthy participants and patients based on the movement pattern of the performed squat exercise, we demonstrate that the model accuracy can be improved when using this augmentation method. In conclusion, this dissertation investigates different aspects of using 3D HPE to analyze movements during physical exercise, starting with the foundation of tracking performance assessment for gait analysis, the use of machine learning to analyze movement patterns for fatigue prediction, and a data augmentation method to balance the class distributions of datasets. With our conducted experiments and developed methods, we aim to advance the field of human motion analysis.