Spatio-temporal feature extraction with a global-local Transformer model for video scene graph generation

In the field of video scene graph generation, spatio-temporal feature extraction and the long-tail effect in relationship classification are core research issues. This paper proposes extracting spatio-temporal features using the global–local Transformer model for video scene graph generation. Methods based on the Transformer architecture and attention mechanism enrich the semantic information of spatio-temporal features in videos, thereby improving the accuracy of relationship classification. In the feature processing module, pose features are introduced to strengthen the semantic representation of objects. In the spatial feature encoding module, a local spatial visibility matrix based on bounding boxes and key points of human pose features is proposed to add the issue of insufficient attention to local details in traditional Transformer encoders. In the temporal feature encoding module, a global random frame extraction strategy is proposed, which considers global temporal features while also taking computational complexity into account. In the relation classification module, to address the uneven distribution of object and relation categories in the Action Genome dataset, a relation classification loss function based on bipartite graph matching and Focal Loss is proposed, which alleviates the long-tail effect in relation classification and improves the accuracy.