ST-CLIP: Learning SpatioTemporal CLIP with GraphTransformer for Action Recognition

Large pretrained models applied to image data have showcased significant progress in visual representation learning. However, the direct application of pretrained image models to video data is insufficient for capturing spatiotemporal context. The challenge lies in effectively transferring image-modal knowledge from web-scale data to the video domain. To enable spatiotemporal modeling, we propose SpatioTemporal CLIP (ST-CLIP) that incorporates a lightweight graph reasoning unit into each encoder of the vision Transformer to efficiently reason about temporal relations between frames. Extensive experiments and ablation studies highlight the competitiveness of ST-CLIP compared to state-of-the-art methods. The proposed model achieves impressive performance on fully-supervised experiments on Kinetics-400.