Human Action Recognition by Invisible Sensing with the Constraint of Privacy Preservation

Human action recognition (HAR) based on visible light videos is prone to expose users privacy. Infrared videos and radio signals are well known invisible sensing approaches for HAR with the constraint of privacy preservation. However, either of them holds some inherent weaknesses, which strongly hinder the HAR performance improvement. Considering the complementarity of the different modal data, we suggest to explore both infrared and radio clues to improve the performance of HAR. Specifically, we design a two-branch heterogenous deep neural network to extract IR and WiFi features respectively. Rather than traditional methods, we propose a deep feature fusion approach which nonlinearly projects IR and WiFi features into a common subspace to maximize the correlation between two modalities. To validate our novel ideas, we build a human action dataset comprising IR and WiFi samples. Based on this dataset, a series of elaborate experiments were carried out, which prove that multi modal sensing can significantly improve HAR performance with the constraint of privacy preservation. The proposed feature fusion strategy outperforms other state-of-the-art methods obviously in experiments.