Temporal refinement with channel topology attention network for skeleton-based action recognition

Skeletal data, which are high-level features of the human body, can directly represent the overall posture and dynamic information, reducing background noise. Most existing studies employ graph convolutional networks. However, this approach not only fails to capture the dependencies between joints that are unnaturally connected in the predefined skeletal graph but also focuses on modeling spatial features. To address these challenges, we propose an innovative 3D convolutional neural network, named temporal refinement with channel topology attention network (TRCTA-Net). This network utilizes multi-scale 3D spatiotemporal convolutions on 3D heatmaps stacked along the temporal dimension. In addition, TRCTA-Net is designed to detect subtle intra- and inter-frame variations of distant joints. TRCTA-Net introduces two key modules: the channel topology attention (CTA) module, which extracts spatial topological features and establishes connectivity and dependencies among distant joints across channels, dynamically learning a channel topology attention matrix; and the temporal topology refinement (TTR) module, which identifies subtle intra- and inter-frame differences, refining joint topological features based on dependency relationships within the local feature maps. We conducted experiments on three datasets: FineGYM, NTU RGB + D 60, and HMDB-51. Compared with previous methods based on GCN and CNN, TRCTA-Net achieves competitive performance without increasing computational cost.