Semantic-Driven Video Question Answering with Fine-Grained Spatio-Temporal Interaction

Research has been conducted on a critical issue in Video Question Answering (VideoQA): how to effectively capture spatiotemporal information from videos for accurate reasoning while understanding natural language questions. Previous studies predominantly employed pretraining networks for images and actions to extract global features. However, due to the lack of fine-grained features, VideoQA performance was constrained. To address this limitation, object pretraining networks were introduced to capture fine-grained spatiotemporal features by extracting object features from video frames. Nevertheless, the current approach still neglects sufficient consideration of semantic interactions between objects and actions in videos, limiting spatiotemporal reasoning capabilities. To overcome this challenge, a semantic-driven fine-grained spatiotemporal interaction approach for VideoQA has been proposed. This method significantly enhances VideoQA performance by supplementing semantic interaction information between objects and actions with image description information. Additionally, to effectively address cross-modal semantic alignment issues, a Bottleneck Transformer has been introduced. It achieves the fusion of different modality features through learnable parameters and backpropagation, bringing new perspectives and methods to the field of VideoQA.