Swin PoseFormer: A Human Activity Recognition Model Using Skeleton Heatmaps

Human Activity Recognition has undergone significant advancements lately. Dynamic skeletal data, represented as the 2D/3D coordinates of human joints, has been widely studied for human action recognition due to its high-level semantic information and environmental robustness. Many skeleton-based action recognition methods adopt graph convolutional networks to extract features from human skeleton data. Recently, Transformer models have shown great success in modeling long-range interactions. Existing research has shown that self-attention mechanism is promising in solving video processing tasks. In this paper, we propose a novel network, Swin PoseFormer, which relies on Swin Transformer backbone to process a 3D heatmap stack instead of a graph sequence as the base representation of human skeletons. We test our Swin Poseformer model on the FineGym dataset and demonstrate that our model outperforms the state of the art model with top-1 and top-5 accuracy of 84.32% and 98.45% respectively.