KiRV: Robust Human Identification via Multimodal Learning Based on Kinetic Gait Features of Radar and Vision

Gait is an appealing biometric pattern that aims to identify individuals based on the way they walk. Gait recognition, a passive human identification technology utilized from a distance without subject cooperation, plays a considerable role in life monitoring, crime prevention, security guarantee, and other identity recognition applications. Although vision-based methods dominate the state-of-the-art field, their performance degrades under poor illumination. In contrast, radar signals are not affected by light and are more sensitive to micro-motion information. In this article, we design a Kinetic feature-based Radar-Vision fused (KiRV) gait recognition method, which leverages millimeter-wave radar echo signals and a video for illumination robust human identification. In the KiRV, we propose a novel kinetic gait feature representation framework based on radar micro-Doppler and visual optical flow information, which are the direct expressions of the gait motion process. The physical meaning of the kinetic features under the two modalities is similar, while the semantic information is complementary. Therefore, the two features can be effectively fused. To learn robust gait information, we propose two 2-D residual CNN-based lightweight backbone networks to encode the kinetic features, respectively, and further propose a two-stream cross-correlated fusion method, including radar-vision cross-correlated fusion (RVCF) and radar-vision gate unit (RVGU) modules. The RVCF adaptively adjusts the attention to radar and vision for better recognition performance, while the RVGU controls the contribution of each modality to the fused feature to improve the robustness of the model. Finally, the gait retrieval task can be achieved through the above innovative model and joint loss calculation at different feature levels. Extensive experiments are conducted in the real world and semi-simulation, demonstrating that the KiRV outperforms state-of-the-art gait recognition methods with well-illumination robustness.