RGSA: Sensor-Based Group Activity Recognition Model With Relation Gating and Spatiotemporal Attention

Sensor-based group activity recognition (GAR) is a challenging task that requires handling complex individual actions and inter-individual relations. An imperative exists to explore a relation learning framework for effectively modeling and dynamically extracting group-relevant actions and interactions in group activities. To solve this issue, we present RGSA, a novel model for sensor-based GAR with relation gating and spatiotemporal attention. RGSA employs a graph structure to model individual interactions achieving dynamic updating of the group relation graph through the graph-passing process. By introducing a relation gating mechanism and proposing the interaction feature correlation reward, RGSA selects group-relevant relations through deep reinforcement learning. Subsequently, an enhanced set of relations is obtained. Additionally, RGSA introduces a temporal attention mechanism. This mechanism assesses the coherence of actions by calculating the consistency between an individual action at a specific timestep and the entire group activity. It quantifies the contribution of the action at that timestep, helping to exclude the interference of irrelevant actions. Meanwhile, RGSA introduces a spatial attention mechanism that selectively integrates and stores interactional information between neighboring individuals based on the contribution of their interactions. It facilitates a more comprehensive capture of the dynamic features of group activity. Experiments are conducted on two datasets, revealing that RGSA proficiently identifies group activities while effectively suppressing interference from non-correlated relations and irrelevant actions in GAR. It ensures a sustained high accuracy in recognizing group activities even in the presence of interference.