mmHAT: 3D Human Arm Tracking with Joint Learning using Dynamic mmWave Point Cloud

Tracking the human arm is essential for a variety of applications, including medical rehabilitation, sports analysis, and human-computer interaction. Current vision-based and wearable sensor-based approaches either struggle with occlusion, poor lighting conditions, and privacy concerns or result in intrusive user experiences. This paper introduces mmHAT, a novel 3D arm trajectory tracking method using mmWave radar. mmHAT proposes an end-to-end neural network design to address two major challenges: the lack of arm semantic information and dynamic variations in the mmWave point clouds. Firstly, mmHAT incorporates a multi-task joint learning framework, where the primary task is 3D arm tracking and the auxiliary task is gesture recognition. This aims to leverage the auxiliary task to guide the network in developing a deeper understanding of the user's arm movements. Secondly, for dynamic mmWave point clouds, mmHAT incorporates a new spatial-temporal feature encoder that aggregates the features of the arms point cloud from a global perspective. We collect ~320K frames of daily arm activity data for experimental validation. The results show that mmHAT achieves an average joint location error of 1.67 cm and angle estimation error of 4.23° for arm joints (i.e., elbow, wrist), while delivering excellent performance with only 2.67 ms latency.