A Heatmap-Based Weighted Multi-stream Fusion Network in Skeleton Modality for Action Recognition

Human Action Recognition (HAR) has received widespread attention in recent years. For skeleton modality, the representation of spatio-temporal motion and the weight allocation of different streams are still under discussion. Therefore, we propose a Heatmap-based Weighted Multi-Stream Fusion Network, HW-MSFN, which attempts to represent spatiotemporal motion in a novel way, prompting the adaptability of 3D-CNN to motion. Additionally, compared with the slow motion obtained frame by frame, we use fast motion which is captured at a larger time scale, in order to attain more general motion characteristics. Finally, to make full use of inter-frame information and intra-frame information, we explore the best weight allocation of multiple stream fusion for skeleton modality. Experiments implemented on 3 databases including HMDB51, UCF101 and FineGYM99 show that our model outperforms state-of-the-art methods in single skeleton modality.