mmPrivPose3D: A RaDAR-based approach to privacy-compliant pose estimation and gesture command recognition in human-robot collaboration

Various sensors are employed in dynamic human-robot collaboration manufacturing environments for real-time human pose estimation to improve safety through collision-avoidance systems and gesture command recognition to enhance human-robot interaction. However, the most widely used sensors – RGBD cameras – often underperform under varying lighting and environmental conditions and raise privacy concerns. This paper introduces mmPrivPose3D, a novel system designed to prioritize privacy while performing human pose estimation and gesture command recognition using a 60 GHz industrial Frequency Modulated Continuous Wave (FMCW) RaDAR with a 10 m maximum range and 29 degrees angular resolution. The system employs a parallel architecture including a 3D Convolutional Neural Network (CNN) for pose estimation, which extracts 19 keypoints of the human skeleton, along with a random forest classifier for recognizing gesture commands. The system was trained on a dataset involving ten individuals performing various movements in a human-robot interaction context, including walking in the workspace and hand-waving gestures. Our model demonstrated a low Mean Per Joint Position Error (MPJPE) of 4.8% across keypoints for pose estimation and, for gesture recognition, an accuracy of 96.3% during k-fold cross-validation and 96.2% during inference. mmPrivPose3D has the potential for application in human workspace localization and human-to-robot communication, particularly in contexts where privacy is a concern.