Fast Human Action Recognition via Millimeter Wave Radar Point Cloud Sequences Learning

Human action recognition using commercial millimeter wave radar is gaining significant attention in smart elderly care and smart homes. Due to privacy concerns, the sensing data often needs to be processed locally on embedded systems with restricted computational resources, necessitating a balance between recognition accuracy and efficiency. In this paper, we propose a fast human action recognition framework based on 3D point cloud sequences generated by commercial 4D millimeter wave imaging radar systems. The framework comprises two primary phases: data preprocessing and spatial-temporal feature extraction. During the data preprocessing phase, we employ a sliding window approach for frame fusion to enhance the spatial information of the sparse point cloud while retaining its temporal features. Additionally, Morton coding is used to address the disorderliness in the point cloud sequence. For spatial-temporal feature extraction, we introduce an innovative two-stage algorithm. In the spatial feature extraction stage, we initially extract local spatial features for each point, utilizing self-attention to construct a local graph and circumvent the limitations of using Euclidean distance in sparse point clouds. Subsequently, 3D frame fusion convolution is applied to extract spatial features at the frame level, reducing the length of the spatial feature map sequence and lowering computational requirements for subsequent temporal feature extraction. In the temporal feature extraction stage, we employ a modified Transformer encoder with fine-grained feature fusion to extract temporal features. We conducted comprehensive experiments using both our collected dataset and the open dataset RadHar. The experimental outcomes demonstrate that our framework not only improves inference accuracy but also maintains satisfactory real-time performance on embedded platforms with constrained computational resources. When compared with state-of-the-art (SOTA) methods, our framework significantly enhances inference speed while retaining competitive inference accuracy. Codes and dataset are available at https://github.com/Feiyuyu0503/FastHAR.