Self-supervised video representation learning based on foreground and temporal information

Aiming at the problem that action recognition models require large-scale datasets for pre-training, this paper proposes an action recognition method based on self-supervised representation. In order to make the network focus on the foreground moving subject and the relative relationship between foreground and background, a self-supervised proxy task based on moving subject is designed. In order to make the network focus on temporal context information and motion trend, a self-supervised proxy task based on temporal information is designed. Firstly, the network is pre-trained in the self-supervised branch, and then migrated to the action recognition dataset for training, so as to obtain the action recognition model. The experimental results show that when the backbone network is ResNet 3D, the accuracy in UCF101 and HDMB51 datasets can be increased by 0.252 and 0.117 respectively compared with random initialization training.