Temporal and Joint Topology-Refined Hypergraph Convolution Networks for Skeletal Action Recognition

Graph convolutional networks (GCNs) have been widely used and achieved remarkable results in skeleton-based action recognition. Due to the coordination of multiple joints required for human actions in real-life scenarios, there may be significant connection distances among these joints in the human body graph structure. we introduce Temporal and Joint Topology-Refined Hypergraph Convolution Network (TJTR-HGCN) to learn diverse topologies and aggregate skeleton features. Leveraging hypergraph theory, TJTR-HGC extends the graph structure beyond physical connections for skeleton-based action recognition. To mitigate noise from unrelated joints, hyperedge structures extract local and global information, capturing spatiotemporal details and higher-order dependencies. It optimizes by designing two modules: Temporal Topology Refinement Hypergraph Convolution (TTR-HGC) learns a shared hypergraph-based topology and refines it with time-specific correlated topology and Joint-wise Topology Refinement Hypergraph Convolution (JTR-HGC). Our approach enhances representational power by transforming graph convolution into hypergraph convolution. Combining the simplified Multi-Scale Temporal Convolutional Network (MS-TCN) with TJTR-HGC enriches temporal feature expressiveness. Furthermore, we apply a new four-way ensemble method that uses relative vectors to the skeleton center as a new data stream to supplement spatial information. Our TJTR-HGCN significantly outperforms state-of-the-art methods on the NTU RGB+D and NTU RGB+D 120 datasets.