Hierarchical contrastive learning for unsupervised 3D action recognition

Unsupervised contrastive 3D action representation learning has made great progress recently. However, most works rely on only the direct instance-level comparison with unreasonable positive/negative constraint, which degrades the learning performance . In this paper, we propose a Hierarchical Contrastive Scheme (HCS) for unsupervised skeleton 3D action representation learning, which takes advantage of multi-level contrast. Specifically, we keep the instance-level contrast to draw the different augmentations of the same instance close, targets to learn intra-instance consistency. Then we extend the contrastive objective from individual instances to clusters by enforcing consistency between cluster assignment from different instance of same category, aims at learning inter-instance consistency. Compared with previous methods, HCS enables intra/inter-instance consistency pursuit via multi-level contrast, without inflexible positive/negative constraint, which leads to a more discriminative feature space. Experimental results validate that the proposed framework outperforms the previous state-of-the-art methods on the challenging NTU RGB+D and PKU-MMD datasets. • A novel Hierarchical Contrastive Scheme (HCS) which takes advantage of multi-level contrast is proposed for unsupervised skeleton 3D action representation learning. • Two learning components including the instance-level and cluster-level comparisons are cooperated for the more comprehensive learning. • The contrastive objective is extended from individual instances to clusters, which enables intra/inter-instance consistency learning via multi-level contrast.