Video2mmPoint: Synthesizing mmWave Point Cloud Data From Videos for Gait Recognition

Gait recognition based on millimeter-wave (mmWave) radar is closely integrated with deep learning. However, laborious data collection and annotation processes result in few publicly available mmWave radar gait datasets, limiting practical application. In contrast, video data are easily collectible, leading to the advanced development of vision-based gait recognition. Therefore, we propose a method entitled Video2mmPoint for synthesizing mmWave point cloud data from videos. Specifically, we formulate a universal human body point cloud template and design a point cloud-skeleton matching module to achieve pose transformations. In addition, we attach Doppler features to points based on interframe skeleton information. We use a statistical method to simulate the randomness of position and intensity distribution, thereby reconstructing a single-frame point cloud. Unlike conventional approaches, Video2mmPoinoft capitalizes on the correlation between human body skeletons and radar point clouds, offering better intuitiveness. Furthermore, it synthesizes data based on mathematical statistics, ensuring the authenticity of the point cloud without necessitating the training of new neural networks. To demonstrate its effectiveness, we collect a gait dataset and conduct comprehensive experiments using a gait recognition network. Using our synthetic data for data augmentation, we significantly improve the accuracy of gait recognition. Specifically, the recognition accuracy increases from 98.59% to 99.64%. Reducing the amount of real data to 1/4, 1/8, and 1/16 leads to accuracy increases of 6.26%, 22.96%, and 31.69%, respectively. In cases where real data for some individuals are absent, we improve recognition accuracy from 62.5% to a maximum of 93.48%.