Hypergraph Convolutional Network with Multi-perspective Topology Refinement forSkeleton-based Action Recognition

Since the human skeleton is a natural topological structure,graph convolutional networks(GCNs) are widely used for skeleton-based human action recognition.In recent research,skeleton sequences are represented as spatio-temporal graphs and topology graphs are used to model the correlation between human joints.However,current GCN-based methods only focus on pairwise joint relationships and ignore potential high-order relationships beyond pairwise relationships,leading to underutilization of the graph structure of skeleton data.To solve this problem,this paper proposes the concept of hypergraph to represent potential high-order relationships of joints.Since the high-order relationships of joints within each frame in the skeleton sequence may vary,the model dynamically learns the high-order correlations within each frame with the K-NN method and initialize the hypergraph structure using the high-level representation of joints.This hypergraph structure can better learn the high-order relationships between joints as the hyperedges dynamically adjust with the evolution of joint features.In current hypergraph neural networks,hypergraph convolution transforms the hypergraph into a simple graph using the Laplace's transformation and then performs graph convolution.This method does not fully utilize the characteristics of the hypergraph.The proposed hypergraph convolution method better utilizes the relationship between hyperedges and hypernodes in the hypergraph,performing hyperedge graph convolution on each hyperedge to learn the high-order relationships between joints.The second problem with current GCN-based human action recognition methods is that the topology built by GCNs to represent pairwise joint relationships is not dynamic enough,such as using the same topology for all frames in a sample.To fully explore the dynamic correlation between pairwise joints,the frame-wise topology modeling method is proposed to capture correlation between pairwise joints under different frames and channel-level topology modeling method is proposed to capture correlation between different feature types.Finally,a hypergraph convolution network with multi-perspective topology refinement(HyperMTR-GCN) is developedfor skeleton-based action recognition,which has a significant advantage on the NTU RGB+D and NTU RGB+D 120 datasets.Specifically,it improves by 3.7% on the X-sub benchmark of NTU RGB+D and by 5.7% on the X-sub benchmark of NTU RGB+D 120 compared to 2s-AGCN.