Crowd Activity Recognition Using Dynamic Salient Frames

The saliency of crowd features from each frame of a video frame sequence varies. Although it is possible to randomly select frames and extract crowd features, the representativeness of these features varies. This is one of the important factors affecting crowd activity recognition. A novel method for crowd activity recognition was proposed in the paper, CAR-DD, which selects dynamic salient frames to address this issue. The saliency of frames was determined based on the inter-frame relationships in the video sequence and dynamic frames were dynamically selected. Then individuals were characterized using appearance features. The correlation between individuals was characterized using multi-head self-attention of Transformer as well as crowd variation among temporal frames. The crowd in the video was characterized through spatial correlation and temporal variation features which were used for crowd activity recognition. Experimental results on the Collective Activity dataset demonstrate that, due to the use of dynamic salient frames, the recognition performance of the method in the paper reaches 95.4% when using only RGB modality data. Additionally, the number of dynamic salient frames is an important factor influencing the method's performance.