Advanced Millimeter Wave Radar-Based Human Pose Estimation Enabled by a Deep Learning Neural Network Trained With Optical Motion Capture Ground Truth Data

This paper presents a deep learning-enabled method for human pose estimation using radar target lists, obtained through a low-cost radar system with three transmitters and four receivers in a multiple-input multiple-output setup. We address challenges in previous research that often relied on extracting ground truth poses from RGB data, which are constrained by the need for 3D mapping and vulnerability to occlusions. To overcome these limitations, we utilized optical motion capture, which is widely recognized as the gold standard for precise human motion analysis. We conducted an extensive optical motion capture study involving various recorded movement activities, which resulted in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mmRadPose</i>, a new dataset that enhances existing benchmarks for radar-based pose estimation. This dataset has been made publicly accessible. Building on this approach, we designed an application-tailored radar signal processing chain to generate suitable input for the machine learning algorithm. We further developed an attentional recurrent-based deep learning model, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PntPoseAT</i>, which predicts 24 keypoints of human poses using radar target lists. We employed cross validation to thoroughly evaluate the model. This model surpasses previous approaches and achieves an average mean per-joint position error of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$6.49 \,\mathrm{c}\mathrm{m}$</tex-math></inline-formula> with a standard deviation of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$3.74 \,\mathrm{c}\mathrm{m}$</tex-math></inline-formula> on totally unseen test data. This excellent accuracy of the reconstructed keypoint positions is particularly remarkable when you consider that a very simple radar was used for the measurements. Additionally, we conducted a comprehensive analysis of the model's performance by exploring aspects such as network architecture, the use of long short-term memory versus gated recurrent units, input data selection, and the integration of multi-head self-attention mechanisms.