Human Motion Recognition Based on Videos and Radar Spectrograms in Cross-Target Scenarios

Multi-sensor fusion combines the benefits of each sensor, resulting in a thorough and reliable motion recognition even in challenging measurement environments. Meanwhile, even with the environmental robustness attained through sensor integration, the recognition model continues to face challenges in cross-target scenarios. In summary, the recognition model is consistently trained using the measurement dataset; however, its performance may decline when applied to unfamiliar subjects. This paper highlights this issue and presents a cross-target human motion recognition model for the radar-camera measurement system. We have developed a modal-specific semantic interaction mechanism that allows the feature extractor to recognize different individuals, thereby removing identity information during the feature extraction process. Furthermore, we have also put forward a meta prototypes learning scheme that suitably adjusts the probability distribution to enhance the generalization capability of the recognition model. To emphasize, the proposed model is implemented without altering the primary designed network architecture, indicating that there is no additional computational burden during testing. In comparison with five multi-modal learning algorithms, we have validated the effectiveness of our model, highlighting that it surpasses previous radar-video-based methods by more than 5% in recognition accuracy. Through experiments using public datasets under different dataset conditions, we verified the generalization ability of our model. Ablation studies and additional parameter studies have been conducted, enabling a thorough examination of each design.