GCN Rewarded Reinforcement Learning for Action Segmentation

Action recognition is a very basic topic for human-robot visual interaction. To enable action recognition for real-world applications, at the core is the problem of action segmentation, which aims to segment action slices from long videos. Existing action segmentation approaches rely heavily on hand-crafted techniques, which are limited on accuracy due to the spatio-temporal complexity of actions across videos. To this end, we formulate action segmentation as a Markov decision process rewarded by observing the action recognition results, and build a reinforcement learning framework to adjust the segmenting procedure iteratively according to the rewards. In particular, to perceive the segmented actions for reinforcement learning, we propose a temporal graph convolutional network (GCN) by representing the context relationships of neighboring action slices, whose states are initialized by considering the consistency of actions, and updated by computing their temporal similarity fusing the tIoU and center distance. Convolutional neural network (CNN) backbone features are used as the nodes of GCN, and the edges of GCN are calculated utilizing the context relationships for spatially and temporally representation. By designing an observation and objective module within the reinforcement learning framework based on GCN representations, we finish the network training for. Finally, we conduct extensive experiments on two public datasets, and by comparisons with some state-of-the-art results, our method is demonstrated.