Key Instance-Based Spatio-Temporal Network for Group Activity Recognition

Group activity recognition involves detecting the collective actions performed by a group of individuals, where identifying the key actors and key frames is crucial for understanding the group's behavior. To tackle this challenge, we propose a spatio-temporal reasoning framework that leverages key instances. Our key instance identification module effectively detects key roles and frames from video sequences, while a graph-based reasoning model dynamically aggregates the features of related actors. We extract joint features and RGB features from video sequences, and these are fused using our multi-modal fusion TCT module, which improves the representation power of the original features. To better understand group activity through spatio-temporal correlations, we further utilize an enhanced cross-transformer module for spatio-temporal synchronous reasoning, considering both time and space dimensions. Our method has been tested on two public datasets, showing that it achieves high accuracy and surpasses many state-of-the-art approaches.